Persistence in Squeak
Ah, deliciously punny post title. You’ll see, assuming you make it to the end of this thing… don’t worry, I won’t blame you if you don’t.
Over the last week or so, I’ve been working on a little toy project, partly to fill a need I have, and partly to fiddle around with developing a web application in Seaside and Smalltalk, and specifically the Squeak implementation of Smalltalk.
Now, there are many parts needed to build a functional web-based application. Obviously you need a web server to actually serve the application. You need some sort of language to implement the application in. You need a framework in which to actually build that application (okay, sure, back in the days of the wild west, people built their own, but you’d be a fool to do that today given the plethora of frameworks available which simplify the web development process). And last but not least, in all probability, you need a data persistence solution.
Of course, the first thing that comes to mind when turning ones thoughts to persistence is a good old fashioned relational database, which has been the cornerstone of data persistence for many a decade now. But when one is working in a deeply object-oriented language like Smalltalk, working with a relational database becomes rather cumbersome due to the rather substantial impedance mismatch between relational and object-oriented data modeling. As a result, we as an industry have turned to tools such as automated object-relational mappers (eg, Hibernate, etc) to try and ease the pain of this mismatch, but in general, the results aren’t what I would call pretty.
Which is why, during my first hack at leveraging a persistence solution for my little Seaside application, I decided to try something entirely different: an object-oriented database called Magma. Unfortunately, it didn’t go too well.
On Magma
Magma is a very interesting project. As a persistence solution, it really aims for the same space occupied by Gemstone/S: to act as a completely transparent persistence solution for object-oriented data models. By that I mean the idea is that you hand Magma an object graph, and it persists it to it’s own custom data format on disk. When you pull it back out, Magma reifies parts of the object graph you’re interested in, and when you modify the graph, Magma spots the changes and reflects them back into the persistent store.
Of course, on the face of it, this seems like absolute magic. You simply work with your objects. When you want to persist a change, you just do something like:
session commit: [ model doStuff ].
And voila, everything just, well, works. Of course, persistence is about more than just simple object storage, in that you also need to be able to query the data model, and be able to do so in an efficient manner. To that end, Magma provides a few specialized collection objects, such as the MagmaCollection class, which provide interfaces for applying indexes, querying, sorting, and so forth.
So, on it’s face, Magma looks like a fantastic solution! The transparent persistence model makes it dead easy to manipulate your data model, and you no longer have to jump through all the object-relational modeling hoops that one would normally have to deal with.
But, alas, it’s just not that easy.
Unfortunately, Magma has one serious fault that rules it out for all but the most basic data-driven applications: It’s slow. Additionally, because Magma absolutely requires per-attribute indexes for any collection you want to query, the number of indexes in a data model can grow substantially, particularly in data mining/exploration tools. Worse, Magma steps on a rather nasty performance problem in Squeak whereby large numbers of files in a single directory (as in, thousands) causes the FileDirectory class to bog down… and guess what happens when you create a large number of indexes? That’s right, a lot of files get created in a single directory, and so you get utterly dismal performance when any index is initially opened.
And as if that weren’t enough, in order to really squeeze decent performance out of Magma, you must start tweaking what are called “read strategies”. See, when you start reifying an object graph, you have to make a decision on how deep to go before you stop. After all, if you have a deep tree of objects, unless you plan to traverse that whole tree at some point, it’s a waste of time to load the whole thing all at once. So the “read strategy” dictates at what depth various parts of the object graph are read. But ultimately, what this equates to is deep micromanagement of the database behaviour, and, quite frankly, I have absolutely no interest in that.
Thus, after many days of fighting, I’ve decided to throw out Magma. Which is rather painful, as I already have an object model built up assuming it’s use. Fortunately, the very nature of Magma means you don’t really tailor the object model too tightly to the database, but things do leak through here and there, and the model itself must, to some extent, be designed to facilitate querying, traversals, and so forth. Thus, any movement away from an RDBMS will necessitate rethinking my data model.
A Way Forward?
So what now? Well, I’ve decided to take the hit and switch to a solution based on Glorp, an object-relational mapping system for Smalltalk, and PostgreSQL, that venerable RDBMS. Of course, this will likely come with it’s own issues, first and foremost one of installation…
Unfortunately, while Squeak package management has taken a step forward with Monticello, the management of dependencies between packages, and inconsistencies between platforms (eg, Pharo vs Squeak) means that things are a lot harder for the user than they need to be. In this particular case, the original Glorp port is rather old. So the folks developing SqueakDBX have worked to port over the latest version to Squeak, with some success. Unfortunately, their installation script doesn’t appear to work in Pharo. So I had to resort to pulling in their loader classes and then manually executing the installation steps by hand. Tedious, to say the least.
But, on the bright side, I have a Pharo image that seems to have a functional Postgres client and Glorp install, so I can start fiddling with those tools to see if they can meet my needs.
Which brings me back to the double entendre. Because returning to the Squeak world has reminded me of one thing: Occasionally the tools get in your way as much as they clear it out for you, and so sometimes you really do need to be incredibly… yes, I’m gonna say it, get ready… here it comes… persistent.
Why Developers Should Be Writers
In my many years in the software development industry, not to mention my many years in the software development education industry, I’ve been continually amazed by the tacit acceptance of the fact that many (most?) software developers are terrible writers. The university programmes don’t require anything beyond a simple English 101 class, and companies simply accept the fact that many of their people are, at best, barely literate. It’s a sad, stupid state of affairs, and I figured I’d take a few minutes to explain why I think it’s a detriment to the industry as a whole.
You see, in my mind, at it’s core, software development is fundamentally an act of communication. Of course, there’s the obvious fact that a developer must take their ideas and communicate them to the computer, which then executes them. But as developers, we must also communicate ideas to our users, through the user interfaces we build. And we must also communicate ideas to other developers through the code itself, not to mention the comments therein (after all, as any developer will tell you, development is as much, if not more, about reading code as it is writing it).
Similarly, writing is, obviously, an act of communication. When a writer writes, their goal is to take amorphous, ephemeral ideas, and turn them into concrete, written words which preserve the essence of those ideas and communicates them to the reader.
Now, in order to communicate complex ideas through written word, one must master some very basic skills:
- The ability to clearly conceptualize an idea and transform it into a more concrete expression.
- The ability to break down that idea into simple parts that can be easily explained.
- The ability to explain those parts in a way the reader can understand.
- The ability to take those parts, now explained, and to synthesize them into a coherent whole.
Does this sound anything at all like software development?
Furthermore, a capable writer pays attention to detail. He is as much concerned with the way an idea is expressed as he is with communicating the idea itself. For example, I could’ve written this entire post in short, terse sentences with no paragraph breaks. But I care as much about how these ideas are communicated as I do about the actual act of communicating them.
Similarly, in the area of software development, while two developers may derive the same solution to a problem, one may choose to write terse, difficult to read code that’s poorly formatted and organized, and consequently difficult to maintain, while the other may produce code that’s precisely the opposite.
By now you can probably guess what I’m getting at. I would surmise that you would find a correlation between developers who are skilled writers, and those who produce code that’s clean, readable, and maintainable. Now, that’s not to say there aren’t exceptions. I’m sure there are many many developers out there that are great writers yet terrible developers, and vice versa. But I would contend that, statistically, you would find a correlation between writing skill and development skill, and at their core, these two disciplines are really very similar.
So why is it that we accept such poor writing skill in the development community? Quite honestly, I’m not sure. I think part of the issue is the fundamental belief that software development is an engineering skill, a process that’s dominated purely by technological problems that must be solved with technological solutions. I suspect it’s also driven by a false dichotomy, the idea that writers are “thinkers” and technologists are “doers”. But I truly believe it needs to change. Meanwhile, the next time I interview someone, I may be tempted to ask them to write a short essay on a topic of my choice…
The Seaside Web Framework
While I’m aware that I have, what, maybe two readers of this blog, I thought I might actually start regularly writing a few posts on some of my recent work in the realm of software development. Why? Well, I enjoy writing, and I enjoy… let’s call it “self-gratification”, so posting on my blog seems like a great way to satisfy both of those needs.
So, with all that said, I bring you the kickoff post, covering Seaside.
A Little Introduction
Anyone who’s done any amount of serious web development understands what an absolutely horrible place we, as a development community, find ourselves in. We’re still manually authoring HTML, hacking Javascript, writing AJAX callback hooks by hand, and generally doing all the nasty, gritty, ugly work to make rich web applications possible. Of course, frameworks and abstraction layers have come along to make this a bit easier (Google’s GWT is a great example), but in the end, many of us are still stuck in the dark ages when it comes to web development.
Enter Seaside.
Okay, no, wait, let’s back up one step further.
A Little Pre-Introduction
You all know what Smalltalk is, right? For those not in the know, it’s a nice, high-level, consistent, clean object-oriented programming language that is really the grandfather for many of the programming languages we see today.
Of course, if that were it, we’d probably all be using Smalltalk today. But, alas, the history of Smalltalk is a messy one, sharing many similarities with the Unix battles of old, plagued by myriad, incompatible, expensive implementations that drove away developers to other solutions.
Furthermore, it’s a little strange in at least one respect: rather than code being stored in files, and compiled into binaries, the entire environment, including all your code, is composed into a single “image” from which you must do all you work, including editing, debugging, and so forth. This has great advantages, for example:
- The entire environment is available to you and can be inspected and modified as you desire.
- Deploying an application involves just copying over an image and firing up a VM.
But there’s also major disadvantages:
- You must use the tools provided in the environment (ie, editor, debugger, etc).
- Integration with version control systems isn’t necessarily that great.
- It can be tough to figure out where your code ends and the system begins.
So the picture is certainly mixed. But the sheer power of Smalltalk, the language, and the encompassing environment makes it, at the very least, incredibly intriguing.
As for implementations, for hobbyists, the most commonly used environment is Squeak, or it’s more professional cousin Pharo. I’ve settled on the latter, as it seems to be taking a more professional tack, but it’s really a matter of preference.
By the way, what I’ve said isn’t actually true of GNU Smalltalk, but having never used it, I can’t really speak to it’s viability as a platform. Of course, feel free to take a look at it and let me know what you think!
Where Were We
Oh yeah. Enter Seaside.
So what’s Seaside? Well, it provides an advanced web development framework for Smalltalk that allows the developer to just, you know, get on with it already.
Yeah yeah, I know, you’ve heard that before. So let me illustrate an example for you, and perhaps you’ll see why Seaside excites me so much.
The Example
The program we want to develop is incredibly simple:
- It presents a counter to the user.
- It presents a “decrease” link which lowers the counter.
- It presents an “increase” link which increases the counter.
That’s it. Now imagine, in a traditional web framework, how you would do this. Well, obviously, you need some amount of state, here, in order to track the counter. You could squirrel the value away in a hidden field in a page form (seriously ugly). Or you could assign the user some kind of session ID, and then track the state on the server, using that session ID as a reference (somewhat complicated). Either way, you, the developer, have to focus on how, exactly, that state will be managed.
Now let’s look at how this program would be expressed in Seaside. First, a class declaration:
WAComponent subclass: #Counter instanceVariableNames: 'count' classVariableNames: '' poolDictionaries: '' category: 'Counter'
This is a simple class declaration describing a subclass of WAComponent named Counter, and containing an instance variable called ‘count’. Okay, so now we need an initializer:
Counter>>initialize super initialize. count := 0.
Again, nothing too special here, we just want to initialize our superclass and our counter. But now comes the meat of the program, and the magic:
Counter>>renderContentOn: html html heading: counter. html anchor callback: [ counter := counter + 1 ]; with: 'increase'. html space. html anchor callback: [ counter := counter - 1 ]; with: 'decrease'.
Voila, that’s the entire application, including links and state management.
No, really, that’s it. The whole thing.
So, how does it work? Well, first…
A Bit On Blocks
Like other high-level languages such as Perl, C#, and others, Smalltalk supports the concept of a closure, which is called a block, encapsulating a chunk of code along with it’s lexical scope. That code can then later be invoked at your leisure. For example:
| var block | var := 5. block := [ Transcript show: 'Hello world, my value is '; show: var; cr ].
The variable ‘block’ now contains a reference to a closure which we can then invoke later with:
block value.
This block remembers everything in it’s lexical scope, so, for example, the variable ‘var’ will retain it’s value, 5, and be emitted on the transcript. This fact, that closures are stateful code objects, is key to the way Seaside works.
Back To The Example
So, in Seaside, you never hand-write HTML. There aren’t even any templating languages. You generate all your HTML with code.
Yes, I know, this is weird, but bear with me.
You see, this has a major advantage. Consider the following piece of code from the example:
html anchor callback: [ counter := counter + 1 ]; with: 'increase'.
Of course, this spits out an anchor. Nothing fancy there. But notice how we didn’t specify a URL? That’s weird enough. But notice something else? There’s an argument called ‘callback’, and we’re providing it a block of code. Can you guess what’s happening here?
That’s right. Under the covers, Seaside generates a URL for us. When the link is clicked, Seaside invokes the callback automatically. And because the block remembers the lexical scope, it can fiddle with the counter variable, incrementing it.
So because we let Seaside generate the HTML, suddenly our program is incredibly simple. Under the covers, Seaside manages all our state for us, associating an instance of the Counter object with our browser session. When those links are clicked, the callbacks are invoked in the context of that Counter instance and can manipulate the state of the system. Suddenly we’re no longer hacking HTML, parsing CGI parameters, and all that hideous garbage. We simply write what we want (‘when the user clicks this link, increment the counter’), and Seaside does the rest.
Conclusion
So there you go. A really quick intro to Smalltalk and Seaside. As you can tell, this is incredibly exciting to me. Why? Well, developing web applications has always struck me as incredibly tedious. Rather than just being able to write my damn application, I’m stuck parsing query parameters, managing state, manually handling state transitions, and a whole bunch of other garbage that’s really only peripheral to the actual act of building an application. Seaside, on the other hand, gets rid of all that tedium and lets me focus on the important thing: building a powerful application.
And note, I’ve only just scratched the surface here. Among Seaside’s other powerful features, it has cleanly integrated:
- JQuery
- Prototype
- Scriptaculous
- A general AJAX framework for doing partial page updates
- And probably a whole bunch of other stuff.
Mighty cool if you ask me.
So, all this said, again, the picture isn’t completely rosy. As with all things, there are many issues that Seaside developers must face:
- Myriad persistence solutions that are of mixed quality.
- Code management issues.
- Deployment issues.
- Scaling and performance challenges.
And probably other stuff, too. Which will, of course, be fodder for further posts on this topic.
Git Lesson 2 - Pushing a local repo into SVN
For some time now I’ve been using git as my front end to the Subversion server at work, and I’ve never looked back. And as a result, one of the things I occasionally find myself doing is creating a local git repository in order to manage little side projects I happen to be working on. But, of course, eventually those projects need to be pushed into SVN, and in the process, it’s nice if one can preserve the local commit logs (it’d be trivial to just push the blob of code into SVN and then create a new, local git-svn repo, but that’s not nearly as nice).
Fortunately, git makes this remarkably easy. First, in your git repo, rename trunk so you can get it out of the way:
git branch -m master local
Next, you need to configure your git-svn bridge. My last blog entry on git covers this topic, and it’ll probably look something like this:
git config --add svn-remote.trunk.url svn+ssh://svn/repo/ git config --add svn-remote.trunk.fetch trunk/project:refs/remotes/trunk
Then, fetch the new git-svn bridged repo:
git svn fetch trunk
When you do this, because you don’t have a master, git will kindly create one for you corresponding to the new git-svn bridged branch. Lucky! So now we just need to get the local branch changes into master.
Ah, but there’s some trickery, here. If you were to just do a naive merge from local to master, the root commit on master would end up getting tacked onto the end of the local branch, which is exactly not what we want to happen. The solution is to rebase local to master first:
git checkout local git rebase master
Then you can merge and dcommit:
git checkout master git merge local git svn dcommit
Git will then proceed to push each of your local commits into SVN, and voila, you’re done! Then you can just delete the local branch, as you obviously don’t need it anymore.