UNISA 2014!

UNISA registrations open tomorrow. I’m quite excited to be registering for my third degree. No I won’t be working with Deborah Patta though.

The problem’s that I still haven’t decided what I want to study. Knowing myself, it will likely end up being decided at the casting of lots. For whatever I end up studying, I have a lot of recognisable prior knowledge, but I’m not taking shortcuts, doing all 30 modules.

The idea is to tackle two years’ modules in 2 years, and spread 3rd year over two years. Here’s to not having a life over the next 3-5 years!

rwt.to Update

It was around this time last year that I wrote my board exams, about 52 weeks back. I remember after writing the last exam, thinking that I’m done, and I can now focus on my project. See, it was very tough in the weeks that I was studying because I had to discipline myself to try not to think of my little project.

So, at around this time last year I scrapped all the code that I had worked on, and started rebuilding rwt.to. It’s been quite a journey! I remember in February thinking that I would have everything done by end of March, but then I went on a lengthy secondment that took a lot of weekend time away from my hands.

I kept on moving my target, like a man reaching for the moon.

What is rwt-to?

It has a dodge name, at least that’s what I’ve seen some people thinking as I explained what rwt-to is.

rwt-to: an integrated public transit planner based in South Africa. It helps a commuter find their way using public transport, giving them schedule time estimates and cost of trips.

I finally framed the best way to explain it, can I get a “whoop whoop!”

Getting to Know JavaScript

The technology behind rwt-to is JavaScript. I use JavaScript to write all the logic, to interact with the datastore and of course for the interaction on the client-side. As you’d know I use Node.js and MongoDB, a bit of a controversial couple.

When I first started with rwt-to, my JavaScript knowledge was quite sparse. I could do some ‘stuff’ on the DOM, with the help of jQuery anyways, but I realised that I didn’t really know JavaScript. So having spent a year; conceptualising, scribbling and learning to create in JavaScript, I could say that I have certainly learnt a lot. I’m at the point where I’m no longer figuring out how to implement something, but how to implement it efficiently.

I remember getting my search algorithm working in January, it was a night/morning worth all the celebration. I had achieved something that would validate my concept, and give me the strength to continue on with the project over the coming months. Over the months I’ve built on top of that algorithm, a number of features that I believe will make rwt-to a useful service when it’s ready for public use. I can’t share the detail yet, but I’m silently excited!

Node.js – why you in my toolbox?

I remember at the end of July, I set a beta launch target of mid-August, and I worked as hard as I could to meet the target. I stumbled upon a significant blocker, after all the math and crazy stuff I had done to get things working, I still had a bug that I couldn’t chase down and fix.
There’s a night where I spent about 2 hours on the Internet reading up on Node.js and its use cases. It was really a time of severe doubt, because I felt like I had invested 9 months of my life in the wrong tool for the job. Either way, I had to press on. I was at the point of accepting that if things don’t work out, I can always start over with a different language.

although sometimes I come across as a pessimist, I consider myself to be a cautious optimist, a realist

I tend to be realistic when it comes to such things, although sometimes I come across as a pessimist, I consider myself to be a cautious optimist, a realist. At some point that week I considered drawing a timeline for how long it would take to port all my work to good ol’ PHP. However, I still knew that PHP wasn’t the right job for the task at hand. I remember having overflow errors in PHP when my search heap grew out of hand, and that alone reinforced my belief that JS would work as I was managing well with huge heaps. I should disclaim though that it’s likely I was doing something wrong, and I’m not really blaming PHP for that, but I had other issues with it.

I eventually found the solution to my bug, and it had to do with JavaScript referencing. Even though I was passing an object around for manipulation, it wasn’t being cloned, so I was doing all sorts of crazy things to the same object. I’ll illustrate someday anyways. On with the journey I went, and now the moving target was set at early September.

MongoDB flame wars

I must say, I started using MongoDB when it was at 2.2 (at least that’s what I remember). I liked it not because of ‘web scale’, sharding and all the wonderful things </s> that were being said on the Internet about it, but I liked and chose it because of the following:

  1. It had geolocation indexing, which is a no-op when you’re building a location-based project
  2. It promised a dead-simple API for working with arrays. To illustrate, in PHP & MySQL I had to convert an array to a string before sending the query through with the ODM. That’s risky and cumbersome. I’m sure there are more efficient ways of doing it (prepared statements I think), but for someone with little knowledge, it turned me off.
  3. Control. Novice developers using PostgreSQL with PostGIS could maybe agree with me here:
    PostGIS feels like a black box, along with PostgreSQL actually. The software promised all the features that I wanted for geolocation, but I felt uncomfortable with putting my data in a black box and knowing that it just works without having control of certain functions. With Mongo I had to build most functions from scratch, allowing me to understand everything that was happening, and to be able to control how certain functions work.
  4. A query API that I understand. 
    Even though I didn’t know the details of how some DBs work, I can safely say that my SQL is in shape. I can get dirty with SQL, so things like JOINs don’t worry me much. The problem comes when trying to read a long SQL query mushed together with some PHP, at 3 a.m. Headache much? With Mongo, I found that I could quickly read what I wrote and see where I’m going wrong without having to run the query itself.

Instead of complaining about the lack of JOINS, I embraced the art of denormalising data

Having said all that, I do appreciate the work that MongoDB Inc. (10gen) are doing. There’s a lot of criticism about MongoDB, some of it valid, and other just failed people trying to make MongoDB a scapegoat. An occasional story pops up on Hacker News, but I’m lazy to find some links. I RTFM when I started using Mongo, and I understood that I was losing the ability to JOIN, but for me I saw great wealth in what I was gaining as I felt it would work for my edge-case.

12 months later, Mongo is in version 2.4 with 2.6 a few months away. I have GeoJSON, which my algorithm heavily relies on as, and I’ve been able to complete my prototype. In beta testing Mongo is faring well, my data is backed up in the clouds, and I’m happy.

Thanks MongoDB, and Obama!

JavaScript Slow?

Another speed-bump that I came across was with JS performance. As I added data to rwt-to, I noticed performance degradation. An algorithm that once took 5 seconds at most was starting to run over double to triple the time, and sometimes timing out. I got my hands dirty and learnt a bit of profiling. I was able to reduce the algorithm back down to about ~7 seconds.

However, as I added more data again, performance took another hit. As I learnt about algorithm running times, I realised that I had an exhaustive algorithm, and I needed to apply heuristics sooner than I had anticipated. This realisation was sometime in October, so I’ve been working hard when I can to improve things. I currently have a worst case running time of ~40 seconds in production. I have managed to reduce it down to ~20 seconds in development, but it’s still bad.

If I can’t answer the question: “How do I get from Brakpan to Centurion?” in less than 5 seconds, I’d be failing the user of my service.

With that said, I’m working on some exciting stuff that will greatly reduce running time, I had initially targeted 500 milliseconds, but that’s a bit impossible right now. I think 5 seconds is still acceptable considering that Google does it in about 2-3 seconds.

Data, Data, Data!

Taxi Industry

Anyone with ambitions of creating a public transit planner, but hasn’t ran into the taxi industry, should pack up and go back home.

The taxi industry is the core of our public transport in South Africa. I estimate that you can get to over 90% of populated areas using taxis. Most bus services are regional, stuck in the past regarding availability of data, and are similarly unpredictable. There have been great advances with the likes of Rea Vaya and MyCiti, but those cover a small geographic part of the country. The meat is where the taxis are.

I had/have a strategy regarding taxis, but boy was in for a shock when I started implementing it! I’ve had a taxi association official hint to give me information if I bribe him, I’ve had associations not respond to me, threats when approaching taxi officials. I just haven’t gotten the t-shirt yet. There’s major rivalry with some of the taxi associations, so it’s sometimes a bit dangerous asking for data as the associations keep it confidential.

I’m not done yet, in the words of Arnold: “I’ll be back!”

Metrobus

If you live in Joburg, you probably know Metrobus, the unicorn bus service that covers most of the Joburg area, yet not many of us know where all these buses come from and go to. I could safely say that there’s probably only one person in the Southern Hemisphere who knows all the routes and stops.

I’ve had experiences akin to Juggernaut hitting the brick wall in The X-Men. I was told about all these fancy tender processes that I had to follow with Metrobus, procurement policies that I still don’t know of, and later deafening silence. The problem’s that like many government services, you always speak to the acting this and that, and tomorrow another acting this and that has filled the previous acting this and that’s position.

Government is like Hollywood, full of acting roles

It was quite saddening though learning that one of my competitors is working with Metrobus, without them being subjected to the same rubbish that I was fed about processes and whatnots. To be honest, I lost respect for some people in the process.

Yo, Imma let you finish, but I haven’t given up yet!

Metrorail, outdated schedules

Loliwe, Stimela! Anyone have memories of the old Spoornet days? I only have childhood memories 🙂
PRASA has come a long way, but there’s still the apartheid shadow hanging around. I can safely say that a lot of people still view Metrorail as an apartheid legacy transport system, which came as a result of spatial planning back in the day. You don’t see Metrorail slouching into Sandton or Houghton, though I must say that PRASA are probably doing all they can.

The problem with Metrorail is that we don’t have access to their schedules. Officials don’t respond when I ask for information, and the schedules on their website are ancient (though some might still be applicable). I wish Gauteng Metrorail could take a leaf out of Cape Town Metrorail’s book.

Y U no Integrate?

Lastly, we’ve heard lots of talk about government integration of public transport. I still say that it won’t work with the taxi industry, and the odds of success are slimming down further with every new ‘smart card’ system that each provider implements. We can only grab popcorn, wait and enjoy the show.

Recently I saw Rea Vaya bus stops right next to Metrobus stops. I think that’s what we refer to as integration. Millions are being poured onto the Gautrain, which a friend of mine says benefits the few. He is probably right, which is the sad thing cos although I grew up in the ashes of apartheid, what government is doing with their planning seems to be sort of similar to the spatial planning of the days gone.

Ah, before I forget the orange bus service … To cover my back with people who will wonder, here: PUTCO PUTCO PUTCO. In case you Ctrl + F and search for it, I’ve mentioned it thrice.

Just slap the word freedom around, and it’s all post-94. I know, corridors of freedom!

Now now, balance thyself!

My apologies, I am quite frustrated by the suboptimal planning, I don’t mean to criticise our government that much. My apologies if I appear biased. One of the reasons why I’m working on rwt-to is the hope that I’ll be able to lend a hand someday in helping with optimising public transport. Until I get there, I’ll remember that I live in a glass house.

Mzansi, Competition; Innovate!

I was telling the creator of awayto.be (a competitor) a few months back that I feel that SA isn’t producing the best it can. Aside for awayto.be, the current public transit planners lack the most important thing: integration. To be honest, I expected more from CS graduates and engineers, more than something that just piggybacks off querying GTFS data. That’s the reason why I’m continuing to work on rwt-to when there are already public transit planners out there. To bring some competition, a challenge that will benefit commuters in the long run.

I’ve seen a lot of misdirection from the competitors, playing their hands casually in the open, and all the such. A few months ago one of them announced that Metrobus support is ‘coming soon’, while there are other bus services with information publicly available, yet being ignored. I’m still curious as to how that would have worked. As far as I’m aware, only the bus drivers and controllers in Metrobus know what’s still going on out on the streets.

I’ll stop here before I make enemies too soon, again, I live in a glass house. I don’t yet have a solid product to showcase, so I can’t be criticising people who are making things happen.

To Conclude the Sermon

You know when the preacher says he’s almost done, but he’s still got another solid 30 minutes? I’m kinda like that … But let me spare the reader’s eyes and time.

The State of Mobile Apps

Without getting too far ahead of myself, I plan on building apps for the following, likely in that order:

  • Android
  • Nokia (S60) and perhaps Windows Phone
  • BlackBerry, the dying phoenix
  • iOS

Of course, time will tell. I must first complete my optimisations, and complete the design of the website 🙂

The Moving Target

With all the above said, I now expect rwt-to to be functional by April next year. I’ve learnt lessons not to estimate timelines that are too unrealistic. I got burnt on Twitter when my beta launch flopped. Lesson learnt!

Helping Hands

Lastly, I want to say thanks to all the friends who have been helping me out. From beta testers, voices of reason, educators and advisors, and everyone helping with data.

I can’t write a Grammy acceptance speech without first being nominated, so I’ll happily give thanks where it’s due when rwt-to is online.

Church Hopping Canada

Today something reminded me that in everything that happens, I still don’t believe in coincidences. I made peace with not going church hopping in the US, I can always do it at other times. An alternative that I did not think of though is going church hopping in Canada.

Spatially, it’s different to the US in the sense that the province that I’ll be in doesn’t have a lot of towns/cities, and I’ll already be in the capital, which isn’t huge itself. Other places to likely visit might be French states, and far away to go on a Saturday/Sunday in winter. So I started looking on the Internet to see the churches that the city has, and I believe I’ll have enough to keep me hopping around for most of the 10 Sundays there.

So, with that said, I’m officially going to start planning for Church Hopping Canada 2014!

How Dumb? Corner Fred and Harry

I haven’t had a chance to go pick up something that I purchased online two weeks back, it’s currently at the Strubensvalei Post Office. I thought I’d go pick it up this morning, but somehow I struggled to find the location of the post office. Google Maps pointed me to the correct place, but I was skeptical seeing how some POIs are sometimes not at their correct locations.

I took on to the internet to find the address, and I kept getting Corner Fred and Harry Street. The ‘official’ address from postoffice.co.za was “corner Vrede and Harry Streets”. Problem is, there is neither Fred/Vrede nor Harry Street in Strubensvalei. I went back to Google Maps, and after a minute of looking I just thought:

These idiots, the post office is on corner Fredenharry, not Fred and Harry!

As the politicians always speak, heads must roll over this mistake! I ended up not picking up my package as I was becoming late for work.

Where is the Love? – Black Eyed Peas

https://soundcloud.com/abzal/black-eyed-peas-where-is-the-love

It just ain’t the same, old ways have changed, new ways are strange.

Brings back memories 🙂

I feel the weight of the world on my shoulders, as I grow older ya’ll people grow colder. Most of us only care about money making, selfishness got us following the wrong direction.

Google Play Music Coming to South Africa

Android users, Google fans, rejoice! I just discovered something on Google Play Music 🙂

We now have local South African music on Google Play. It’s only available through All Access for now, but it’s all there! I searched for a few Kwaito and Hip Hop artists, and gospel of course lol.

I think that it’s only a matter of weeks before Google announce availability.

For example, I searched for We Will Worship, and found their latest album from this year. Here’s evidence:

google-paly-music

 

Again, the music is now only available on All Access, trying to buy an album will result in the fugly default 404 page:

music-unavailable

I’ve been hoping for this for a while now. I’ve found it a bit off that I’d go to iTunes to buy music, then have it upload to Google servers. I just hope that they don’t price their music in the same way as the US, because we all know that iTunes is cheaper in SA than in the US.

Here’s to waiting for Android Police to tip us on the announcement 🙂