Posts in category 'hacking'
Fun with Open Data
A Simple Problem
So, recently I found myself struggling with a very simple problem: I kept missing our darn garbage pickup date. Yeah, sure, the city sends out a little printed schedule, and it usually makes it to my refrigerator, but given that requires I actually pay attention to the silly thing, it doesn’t end up helping me much.
Now, in the past, I’ve solved this problem by manually punching the pickup dates into Google Calendar. This works, but it’s tedious, error prone, and whenever the new schedule comes out, I have to do it all over again. And if it wasn’t clear, I’m lazy. Real lazy. So last time around I simply didn’t get around to doing it.
And so I miss the pickup dates.
Obviously what I really wanted was an iCalendar formatted file that I could just subscribe to in Google Calendar, at which point I could move on with my life. But, alas, to my knowledge no such resource exists.
An Idea Is Formed
Well, I found myself discussing this with my officemate, Steve, and he pointed out that the pickup schedule is, in fact, available online in raw form as part of the City of Edmonton’s Open Data Catalog. The Open Data Catalogue is a remarkable resource. In it you can find a staggering amount of data about the city, provided in a simple, machine-readable form, and browseable online. It’s really very cool, and it feels like a resource just waiting to be tapped.
Anyway, back to the topic at hand, it turns out the pickup schedule is available right in the catalogue, albeit in a fairly raw form. At this point, the answer seemed obvious: write a tool which could consume this data, and produce an iCal file as output. I could then subscribe to the generated calendar, and voila! No more missed garbage days!
Why Can’t Things Be Easy?
Unfortunately, things got a little tricky once we started digging into the data. It turns out that the city is subdivided into a series of zones. Each of these zones then has one or more garbage pickup days, and so one’s individual pickup schedule varies based on your geographic location in the city. As such, the pickup schedule data is organized as a simple table as follows:
Zone Day Date So, now we have a problem: how do we determine the zone and day for a given household?
Well, it turns out the city provides the zone/day data as a geographic overlay, in the form of a KML data file. KML is a rather complex XML dialect which can be used to represent geographic data, and is used in, among other placed, Google Maps, Google Earth, and so forth. But, it’s a standard, and there are standard tools for handling it, so a gold star for the city!
Alright, so now we have the geographic data, we just need to parse that file, and then based on household location, identify the appropriate zone and day, and extract the correct schedule information. This should be easy…
A Solution
The first question was one of language. I knew I needed a few things:
- Access to a decent web framework, so I could quickly slap together a (preferably REST) web service.
- Something to parse the KML data.
- Something to do point-in-polygon tests, so that, given the boundaries of a zone and a household lat/long, I could determine if the household was in that zone.
- A CSV library would be handy (this is the form the schedule data takes).
- A library for generating iCalendar files.
And, in addition, given this was a quick learning project, I figured it’d be nice to choose something I haven’t written much code in. In the end, I settled on Python, and in particular:
- CherryPy - A simple, lightweight web framework with support for REST built in.
- libkml - A KML library with (poorly documented) Python bindings.
- RosettaCode RayCasting Algorithm - Code that implements the standard raycasting point-in-polygon test (I could’ve written this myself, but… why?)
- Python’s standard CSV module.
- vobject - A python library for parsing and generating iCalendar files.
Now, it turns out the hardest bit was in ingesting the KML file. The libkml bindings, while functional, are awful. They aren’t terribly pythonic, they’re horribly documented, and the examples and tests provide no coverage of the API. As a result, I ended up reading a lot of SWIG binding definition files, in order to determine the method calls available in the object model provided. But, eventually, I was able to figure out how to extract both the zone metadata (name, day, etc), and the polygon definitions, and with that, it was fairly easy to write a small library to identify a household zone based on their latitude and longitude.
After that, the rest was really pretty easy: the code determines the user’s zone, downloads the schedule data in CSV form, identifies the rows for that household, and then spits out an iCalendar file containing that schedule data. Voila, done! The service is available here:
http://api.b-ark.ca/garbage/schedule?latitude=&longitude=
Just enter your lat/long, and you should get a valid iCal file back. Nice! And even better, it took me roughly a day to slap the whole thing together.
Getting Fancy
Upon posting about this on Google+, another friend of mine pointed out that it was rather onerous to expect the user to go look up the lat/long of their household in order to use the service. This got me thinking of alternatives… wouldn’t it be great if the user could just provide their address, and the service could do the rest?
Well, happy days, it turns out Google offers a free REST API for geocoding (the technical term for the process of mapping an address to a latitude/longitude pair (or vice versa)). Just hit the service with a street address, and they’ll hand you back geographic data in the form of a JSON or XML document. Nice!
About ten lines of code later (with the help of Python’s minidom module), I had a service which could take an address, get the lat/long automatically, and return the appropriate iCal file. You can hit this version here:
http://api.b-ark.ca/garbage/schedule?address=
Just append your address to the URL, and you could get an iCal file back. Thanks Google!
Update:
BTW, if you want to take a look at the (quite simple) code, you can check it out on github.
Hacking the Playbook - Redux
Well, I know I said I was gonna write some posts on the Playbook as I begin developing for it. Unfortunately, there are a number of things which have deeply turned me off of the prospect…
Incomplete APIs
Yes, I understand the device isn’t ready yet. I understand the simulators are in beta, and the SDKs are being completed. But the startling omissions in the current APIs make me seriously wonder about the device and the BB developers:
- No text box control. There’s a text field, but no multi-line control. So much for a note taking application or anything similar.
- No date picker. Seriously. Wow.
- No localization support. This is supposed to be an enterprise-level device, and it doesn’t have a localization API yet??
- No rotation support. Just… unbelievable. This is a tablet, ffs. How can they not have landscape/portrait mode available? Hell, apparently no one has even seen a sample Blackberry app that does portrait mode.
- No webkit engine API. They’re “working on it”, apparently. The device is supposed to be out in a month. I mean, really…
Meanwhile, the simulator doesn’t support things like:
- The camera API
- The multimedia API (you know, the thing that, to quote, is supposed to “differentiate” this device from others on the market).
It’s really quite stunning to me, and makes me wonder what other omissions there are in the application stack.
App World Application Blackholed
I applied for App World three weeks ago. And nothing. Apparently my application is being “reviewed”. Well, I ain’t spending time writing code if I’m not even sure I’ll be able to submit the thing.
Ridiculous App World Fees
Yes, the current submission fees have been waived, but it’ll be $20 per submission to App World once the promotion is over. That means every failed submission, every update, is gonna cost $20. It’s ridiculous.
Unprofessional Webinars
The webinars BB posted were, frankly, terrible. The BB consultant running them is brutal, the material is superficial at best, his delivery is moronic, repetative, and frankly, boring… they’re just bad. Meanwhile, they’re full of glaring holes, bad examples, and don’t get me started on the marketspeak.
Meanwhile, every other question seems to highlight another gap in the SDK or simulator… the number of times I heard “we’re still working on a story for that” was impressive, to say the least.
In Conclusion
Everything I’ve seen suggests this device is half-baked at best. Incomplete APIs, crappy presentations, an application process that seems to have stalled out on me, and a fee structure that seems designed to turn away smaller developers… for a $500 device, it really doesn’t seem to be worth the aggravation.
In The Hopper
Yes, believe it or not, I have a hopper. I never empty the hopper, but the hopper exists, and there are items in it. Usually the hopper is overridden with work concerns, be it actual, day-job work, or the consulting stuff I do on the side. Or it may be overriden by more immediate, viceral concerns: sleeping, eating, and so forth. But given it’s the Christmas season and things are slowing down a bit, it’s possible… just possible… I might be able to work on some items from the hopper.
First on the list? Earn myself a Blackberry PlayBook. They currently have a very simple developer contest: Get an application approved and published in their app store by February first, and you get a free PlayBook. A free $500 tablet? Sounds good to me! I’ve already got the SDK installed (the first SDK released is based on Adobe’s AIR platform), and the simulator up and running, so now I just need to start coding. The current plan is a recipe manager application… ‘course, I just need to decide how to present such an application on a tablet device. Fun!
And the second item? Update savsender to build against the latest devkitarm. And while I’m at it, get the code up on github. This was triggered by an email I received back at the end of October asking for an update, and given how simple the codebase is, it’s a request I should be able to fulfill during my spare time.
So, here’s hoping this is a productive Christmas! Assuming I don’t just sleep my way through it…
Hacking the PlayBook
So, hopefully here begins a series of posts on coding for the Blackberry PlayBook. In this first post, I’ll talk a bit about the PlayBook itself, and the SDK currently available.
The Platform
Unlike previous Blackberry hardware, which has traditionally run their custom embedded Blackberry OS, the PlayBook is the first BB device to run their new OS based on QNX, a realtime Unix variant that’s been around for nearly 30 years, now. My guess, based on various reading, is that Blackberry feels this new platform will scale better on the new embedded hardware that’s hitting the market, particularly the multicore devices everyone is anticipating.
As such, they plan to offer three different SDKs:
- A Java SDK, which I presume will offer compatibility with their existing Java kit on BBOS.
- A native QNX SDK, so you can write bare metal applications, such as games.
- An Adobe AIR SDK.
As of this writing, the AIR SDK is the only platform available, a move that, I suspect, reflects a desire, by Blackberry, to migrate developers to that environment (and more importantly, reflects Adobe’s goal of becoming the application development platform for the mobile space).
The SDK
On Linux, the kit comes as a number of parts:
- Flex Mobile, a cross-platform kit for developing Flex applications running on the AIR runtime.
- Blackberry’s Flex kit, which contains the various Flex libraries needed to build PlayBook applications.
- A simulator, which comes in the form of an ISO which is meant to be coupled with VMWare Player.
Of course, if you live in a Windows or MacOS platform, you also get to take advantage of Flex Builder, but alas, it isn’t available for Linux. That said, since the GUI designer tools in Flex Builder don’t work with the PlayBook GUI elements, it’s utility is far more limited, mainly to providing integrated help, code completion, and so forth, none of which is strictly necessary to developing for the PlayBook.
As for Flex/AIR itself, the language is ECMAScript, which is compiled down to SWF’s, which are then packaged up by the Blackberry toolchain, and can then be installed on the simulator.
Up Next
In my next post, I’ll cover compiling the venerable HelloWorld sample on Linux, and getting it running on the PlayBook; while this is actually quite easy to do with the command-line tools, it took a little bit of hunting around to find the proper incantations to make it happen.
After that, I’ll do a little work to dissect the HelloWorld app and write a bit about my first impressions on the kit.
Interesting Stuff! Stay tuned for more Tales Of Interest!