It was around this time last year that I wrote my board exams, about 52 weeks back. I remember after writing the last exam, thinking that I’m done, and I can now focus on my project. See, it was very tough in the weeks that I was studying because I had to discipline myself to try not to think of my little project.
So, at around this time last year I scrapped all the code that I had worked on, and started rebuilding rwt.to. It’s been quite a journey! I remember in February thinking that I would have everything done by end of March, but then I went on a lengthy secondment that took a lot of weekend time away from my hands.
I kept on moving my target, like a man reaching for the moon.
What is rwt-to?
It has a dodge name, at least that’s what I’ve seen some people thinking as I explained what rwt-to is.
rwt-to: an integrated public transit planner based in South Africa. It helps a commuter find their way using public transport, giving them schedule time estimates and cost of trips.
I finally framed the best way to explain it, can I get a “whoop whoop!”
I remember getting my search algorithm working in January, it was a night/morning worth all the celebration. I had achieved something that would validate my concept, and give me the strength to continue on with the project over the coming months. Over the months I’ve built on top of that algorithm, a number of features that I believe will make rwt-to a useful service when it’s ready for public use. I can’t share the detail yet, but I’m silently excited!
Node.js – why you in my toolbox?
I remember at the end of July, I set a beta launch target of mid-August, and I worked as hard as I could to meet the target. I stumbled upon a significant blocker, after all the math and crazy stuff I had done to get things working, I still had a bug that I couldn’t chase down and fix.
There’s a night where I spent about 2 hours on the Internet reading up on Node.js and its use cases. It was really a time of severe doubt, because I felt like I had invested 9 months of my life in the wrong tool for the job. Either way, I had to press on. I was at the point of accepting that if things don’t work out, I can always start over with a different language.
although sometimes I come across as a pessimist, I consider myself to be a cautious optimist, a realist
I tend to be realistic when it comes to such things, although sometimes I come across as a pessimist, I consider myself to be a cautious optimist, a realist. At some point that week I considered drawing a timeline for how long it would take to port all my work to good ol’ PHP. However, I still knew that PHP wasn’t the right job for the task at hand. I remember having overflow errors in PHP when my search heap grew out of hand, and that alone reinforced my belief that JS would work as I was managing well with huge heaps. I should disclaim though that it’s likely I was doing something wrong, and I’m not really blaming PHP for that, but I had other issues with it.
MongoDB flame wars
I must say, I started using MongoDB when it was at 2.2 (at least that’s what I remember). I liked it not because of ‘web scale’, sharding and all the wonderful things </s> that were being said on the Internet about it, but I liked and chose it because of the following:
- It had geolocation indexing, which is a no-op when you’re building a location-based project
- It promised a dead-simple API for working with arrays. To illustrate, in PHP & MySQL I had to convert an array to a string before sending the query through with the ODM. That’s risky and cumbersome. I’m sure there are more efficient ways of doing it (prepared statements I think), but for someone with little knowledge, it turned me off.
- Control. Novice developers using PostgreSQL with PostGIS could maybe agree with me here:
PostGIS feels like a black box, along with PostgreSQL actually. The software promised all the features that I wanted for geolocation, but I felt uncomfortable with putting my data in a black box and knowing that it just works without having control of certain functions. With Mongo I had to build most functions from scratch, allowing me to understand everything that was happening, and to be able to control how certain functions work.
- A query API that I understand.
Even though I didn’t know the details of how some DBs work, I can safely say that my SQL is in shape. I can get dirty with SQL, so things like JOINs don’t worry me much. The problem comes when trying to read a long SQL query mushed together with some PHP, at 3 a.m. Headache much? With Mongo, I found that I could quickly read what I wrote and see where I’m going wrong without having to run the query itself.
Instead of complaining about the lack of JOINS, I embraced the art of denormalising data
Having said all that, I do appreciate the work that MongoDB Inc. (10gen) are doing. There’s a lot of criticism about MongoDB, some of it valid, and other just failed people trying to make MongoDB a scapegoat. An occasional story pops up on Hacker News, but I’m lazy to find some links. I RTFM when I started using Mongo, and I understood that I was losing the ability to JOIN, but for me I saw great wealth in what I was gaining as I felt it would work for my edge-case.
12 months later, Mongo is in version 2.4 with 2.6 a few months away. I have GeoJSON, which my algorithm heavily relies on as, and I’ve been able to complete my prototype. In beta testing Mongo is faring well, my data is backed up in the clouds, and I’m happy.
Thanks MongoDB, and Obama!
Another speed-bump that I came across was with JS performance. As I added data to rwt-to, I noticed performance degradation. An algorithm that once took 5 seconds at most was starting to run over double to triple the time, and sometimes timing out. I got my hands dirty and learnt a bit of profiling. I was able to reduce the algorithm back down to about ~7 seconds.
However, as I added more data again, performance took another hit. As I learnt about algorithm running times, I realised that I had an exhaustive algorithm, and I needed to apply heuristics sooner than I had anticipated. This realisation was sometime in October, so I’ve been working hard when I can to improve things. I currently have a worst case running time of ~40 seconds in production. I have managed to reduce it down to ~20 seconds in development, but it’s still bad.
If I can’t answer the question: “How do I get from Brakpan to Centurion?” in less than 5 seconds, I’d be failing the user of my service.
With that said, I’m working on some exciting stuff that will greatly reduce running time, I had initially targeted 500 milliseconds, but that’s a bit impossible right now. I think 5 seconds is still acceptable considering that Google does it in about 2-3 seconds.
Data, Data, Data!
Anyone with ambitions of creating a public transit planner, but hasn’t ran into the taxi industry, should pack up and go back home.
The taxi industry is the core of our public transport in South Africa. I estimate that you can get to over 90% of populated areas using taxis. Most bus services are regional, stuck in the past regarding availability of data, and are similarly unpredictable. There have been great advances with the likes of Rea Vaya and MyCiti, but those cover a small geographic part of the country. The meat is where the taxis are.
I had/have a strategy regarding taxis, but boy was in for a shock when I started implementing it! I’ve had a taxi association official hint to give me information if I bribe him, I’ve had associations not respond to me, threats when approaching taxi officials. I just haven’t gotten the t-shirt yet. There’s major rivalry with some of the taxi associations, so it’s sometimes a bit dangerous asking for data as the associations keep it confidential.
I’m not done yet, in the words of Arnold: “I’ll be back!”
If you live in Joburg, you probably know Metrobus, the unicorn bus service that covers most of the Joburg area, yet not many of us know where all these buses come from and go to. I could safely say that there’s probably only one person in the Southern Hemisphere who knows all the routes and stops.
I’ve had experiences akin to Juggernaut hitting the brick wall in The X-Men. I was told about all these fancy tender processes that I had to follow with Metrobus, procurement policies that I still don’t know of, and later deafening silence. The problem’s that like many government services, you always speak to the acting this and that, and tomorrow another acting this and that has filled the previous acting this and that’s position.
Government is like Hollywood, full of acting roles
It was quite saddening though learning that one of my competitors is working with Metrobus, without them being subjected to the same rubbish that I was fed about processes and whatnots. To be honest, I lost respect for some people in the process.
Yo, Imma let you finish, but I haven’t given up yet!
Metrorail, outdated schedules
Loliwe, Stimela! Anyone have memories of the old Spoornet days? I only have childhood memories 🙂
PRASA has come a long way, but there’s still the apartheid shadow hanging around. I can safely say that a lot of people still view Metrorail as an apartheid legacy transport system, which came as a result of spatial planning back in the day. You don’t see Metrorail slouching into Sandton or Houghton, though I must say that PRASA are probably doing all they can.
The problem with Metrorail is that we don’t have access to their schedules. Officials don’t respond when I ask for information, and the schedules on their website are ancient (though some might still be applicable). I wish Gauteng Metrorail could take a leaf out of Cape Town Metrorail’s book.
Y U no Integrate?
Lastly, we’ve heard lots of talk about government integration of public transport. I still say that it won’t work with the taxi industry, and the odds of success are slimming down further with every new ‘smart card’ system that each provider implements. We can only grab popcorn, wait and enjoy the show.
Recently I saw Rea Vaya bus stops right next to Metrobus stops. I think that’s what we refer to as integration. Millions are being poured onto the Gautrain, which a friend of mine says benefits the few. He is probably right, which is the sad thing cos although I grew up in the ashes of apartheid, what government is doing with their planning seems to be sort of similar to the spatial planning of the days gone.
Ah, before I forget the orange bus service … To cover my back with people who will wonder, here: PUTCO PUTCO PUTCO. In case you Ctrl + F and search for it, I’ve mentioned it thrice.
Just slap the word freedom around, and it’s all post-94. I know, corridors of freedom!
Now now, balance thyself!
My apologies, I am quite frustrated by the suboptimal planning, I don’t mean to criticise our government that much. My apologies if I appear biased. One of the reasons why I’m working on rwt-to is the hope that I’ll be able to lend a hand someday in helping with optimising public transport. Until I get there, I’ll remember that I live in a glass house.
Mzansi, Competition; Innovate!
I was telling the creator of awayto.be (a competitor) a few months back that I feel that SA isn’t producing the best it can. Aside for awayto.be, the current public transit planners lack the most important thing: integration. To be honest, I expected more from CS graduates and engineers, more than something that just piggybacks off querying GTFS data. That’s the reason why I’m continuing to work on rwt-to when there are already public transit planners out there. To bring some competition, a challenge that will benefit commuters in the long run.
I’ve seen a lot of misdirection from the competitors, playing their hands casually in the open, and all the such. A few months ago one of them announced that Metrobus support is ‘coming soon’, while there are other bus services with information publicly available, yet being ignored. I’m still curious as to how that would have worked. As far as I’m aware, only the bus drivers and controllers in Metrobus know what’s still going on out on the streets.
I’ll stop here before I make enemies too soon, again, I live in a glass house. I don’t yet have a solid product to showcase, so I can’t be criticising people who are making things happen.
To Conclude the Sermon
You know when the preacher says he’s almost done, but he’s still got another solid 30 minutes? I’m kinda like that … But let me spare the reader’s eyes and time.
The State of Mobile Apps
Without getting too far ahead of myself, I plan on building apps for the following, likely in that order:
- Nokia (S60) and perhaps Windows Phone
- BlackBerry, the dying phoenix
Of course, time will tell. I must first complete my optimisations, and complete the design of the website 🙂
The Moving Target
With all the above said, I now expect rwt-to to be functional by April next year. I’ve learnt lessons not to estimate timelines that are too unrealistic. I got burnt on Twitter when my beta launch flopped. Lesson learnt!
Lastly, I want to say thanks to all the friends who have been helping me out. From beta testers, voices of reason, educators and advisors, and everyone helping with data.
I can’t write a Grammy acceptance speech without first being nominated, so I’ll happily give thanks where it’s due when rwt-to is online.