Gears Within Gears

I’ve uploaded the slides from my talk at QCon San Francisco entitled Internal DSLs in Groovy, Ruby and Others. You can find them here (19MB) released under a Creative Commons Attribution-Noncommercial 3.0 license. Over time the talk morphed into a more holistic approach to DSLs that tried to distill general principles from multiple programming languages; hopefully the folks showing up didn’t mind too much that I didn’t cover a lot of ground in Groovy.

It clocks in at around 120 slides but I breezed through them in about 40 minutes, well shorter than my hour slot; next time I’ll take it more slowly and spend a little bit more time on the code samples. It’s hard for any audience to process code at high speed, and I tend to forget that I’ve had all the time in the world to get comfortable with the samples I’m using but the people I’m talking to probably haven’t.

My goal in the talk was to make an argument for the use of these things and try to demonstrate how an internal DSL is different than a standard API. An internal DSL is a kind of API, sure, but one with a specific goal and what can only be described as a kind of distinct parsing layer. It’s a messy but human-friendly interface top of what should remain cleaner code underneath, and it’s handy in spots when the goal of the code is to describe what, not how. An interface that follows strict command-query separation is your enemy here, at least the way I see it.

Many thanks to Dan North and Martin Fowler for the time they spent with me helping to get this ship-shape, but especially to Neal Ford for inviting me out and providing me with his original talk template and just being a generally all-around cool guy. I had a great time and I hope others did as well.

I was re-reading Joel Spolsky’s Field Guide to Developers recently, and while ruminating on his usual mix of striking insight and infuriating nonsense I got to thinking about why it is, exactly, that open team spaces work so well. Joel is right that cubicles suck, and if offices are working for his company then I’m happy for him and for his employees. But he’s limiting his data input in a pretty dramatic way if that’s all they’ve tried. Given a choice between isolating myself from my team or enhancing the effectiveness of my team space I’ll take the latter every time.

Writing successful software isn’t even mostly about cranking out great code, though that part is certainly important. It’s about working with people to build something people want to use, and in order to communicate effectively people need lots of talk bandwidth, preferably of the face-to-face kind. High-functioning teams prefer working in team spaces not because they can’t afford swanky offices, but because team spaces and pair programming are far, far more productive than offices. If you’re starting a company, for goodness sake, don’t spend all your money building silos that just happen to look like offices. Build a great team and keep them together, in the same room, for as long as possible.

Noise is golden

The most important thing you can do to encourage your team to stay together in the same room is to promote a culture of pairing on programming tasks. When I work alone, without a pair, I find it easy to get distracted, and so I try to foster an environment that minimizes distractions. In other words, I use headphones. But I prefer working in pairs, because when I do so I find it much easier to stay on task, and so I try to foster an environment that maximizes team communication. It is the missing piece of the puzzle that makes team rooms not just possible but desirable. Pairing is a great way to work precisely because you don’t need crutches to help you focus, and once you’re freed from that concern it opens you up to other ways of collaborating.

If you’re trying to encourage an environment that allows talented developers to recline in splendid isolation eight hours a day then you’re doing it wrong, and not just because it takes more than developers to build great software. By the same argument that’s typically made for Aeron chairs, isolation is penny-wise and pound-foolish; cacophony may be distracting, but it won’t stop a team dead in its tracks anything like the way silence will. That’s why agile teams participate in morning stand-up meetings even though they generally work in a team room anyway: not because they expect that, on any given morning, they will glean some kind of deep insight about the state of the project, but because simply providing an opportunity for serendipity-a question about a bug, an offhand comment about an upcoming requirement-redounds in surprising ways. We have short boring meetings now so as to avoid long stressful ones, the ones you’ll have six months from now when you figure out what you missed.

Thanks to to Dan North, who gave me valuable feedback on this post: “It’s fine,” he said, “just post the damn thing.”

Desi McAdam and Jim Remsik of Hashrocket recently got featured in a New York Times article on he practice of pair programming. It’s wonderful that they believe so strongly in the practice and devote so much time and attention and money to hiring great people and providing them with a world-class work environment, but I thought Obie’s followup really excludes folks who might otherwise get a lot out of it. Hashrocket’s setup provides an optimum environment for pairing, but the folks who first came up with the practice didn’t have that kind of workspace available everywhere, and you don’t have to, either. Here’s five things I thought Obie missed.

5. Pair Programming Doesn’t Require Expensive Hardware

All it requires is a computer you’re willing to treat as a shared workspace. Ideally it should be as powerful as a standard developer workstation. If you’d like to, use a dedicated account on one pair’s laptop. Many IT departments already have spares of everything else-keyboards, monitors, and mice-locked away in a storage room somewhere.

If your workspace is configured for cubes, see if you can borrow a conference room for a few days, until you can decide if you like it or not. And as for me, I program in pairs all day but my desk chair of choice possesses neither bells nor whistles. As with everything else, a good manager should treat you with respect and provide you with the tools that you need to get your job done. But that doesn’t mean that they owe it to you to rip apart your entire office if you decide you’d like to start trying to pair a little bit more.

4. Everybody Need Not Get Together And Love One Another Right Now

Not everyone has to get along to get a lot out of it. If two people don’t play nice then don’t pair them together all that often. It might be unpleasant for some managers to have to enforce certain basic hygiene practices, and it might lead to some uncomfortable conversations. But it seems unlikely that it would result in a major shift, as most workplaces already do this.

3. Many Businesses Are Happy To Try Pairing

In more than two years working at ThoughtWorks I’ve never been on a project that didn’t consist of 100% pair programming, though that’s not the case for all of our projects. But my experience has been that we treat pairing as simply part of a well-run Agile project and we expect our clients to recognize and respect that, and virtually all of them have come around to the opinion that it’s something they should be encouraging in their own enterprise. As part of those projects I’ve paired many non-ThoughtWorks developers from more conventional IT backgrounds and I’m proud to have worked with all of them.

2. Many Software Shops Care Deeply About Return On Investment

This isn’t exactly contradicting Obie’s point. But whereas not every IT shop might care about excellence per se (although I think he might be selling some folks short) they all, from the smallest boutique shop up to largest enterprise business unit, care about spending their IT money wisely. Pair programming makes good business sense: it cuts down on defects, raises productivity and helps you spread technical knowledge radically faster than ever before.

1. Pair Programming Is Not Just For The Elite

You do not have to be an incredibly hardcore developer to enjoy it or want to try it. It will make most developers, whether they be great or small, young or old, or male or female, happier and more productive. If you tried it and decided it wasn’t for you, then fair enough. But everyone a little bit of curiosity-especially those who think it couldn’t possibly work for them-should give it a shot. It’s just a nicer way to work.

I met a hacker from an antique land
Who said: Two tall and heavy mounts of steel
Lie in a basement. Near them on a stand,
Recessed, a dark CRT lies, whose peel’d
Cracked shell of dullest beige, and blinkenlights,
Tell that its fact’ry well those old specs read
Which yet survive, inked on the lifeless thing,
The die that stamp’d them and the power that fed.
And on the burned-in screen these words appear:
“My name is Awsymandias, king of kings:
Look on my racks, ye Mighty, and despair!”
No bits at all remain. Not far away
A data center waits, its humming air
Host to a boundless cloud by th’hour to pay.

Awsymandias is a Ruby library designed to help you set up and tear down complicated (mighty!) AWS EC2 environments and to track running costs over time. It does this by persisting instance-specific metadata, such as (for example) its role within your stack, to SimpleDB, allowing you to deploy your Rails app to the same place from multiple different machines. This is helpful if you want to maintain several different deployment environments, such as testing, UAT, and preview, to be able to deploy to them from any machine with Capistrano and your deploy script available, and to be able to raise them up and tear them down at will.

Wherever possible, it also attempts to extend and improve upon the existing Ruby EC2 gem by bootstrapping a real domain model on top of the hash-like XML returned by EC2’s REST API. Currently this is being done with some funky ActiveResource sorcery which I hope to improve upon in the future.

The gem is available now from Github.

(Apologies to Percy Bysshe Shelley’s original sonnet. Die-hards will please note that it scans and attempts to follow the original meter and rhyme scheme wherever possible.)

A couple of days ago Ola Bini pulled my proposed ISpec mocking framework into Ioke (and did me the kindness of porting a change I made to Ioke/J into Ioke/C# – Number Infinity). It clocks it at around 300 lines of code (twice as many of test code) and I’m pretty pleased with the results, although several aspects of it could bear some refactoring.

Syntax

In its simplest form, it allows you to add simple mocks and stubs by method name:

For more BDD-friendly syntax, it’s expected that you use should receive syntax, which supports more advanced mock creation constructs:

The following method call count assertions are supported:

• never
• once
• twice
• atLeastOnce
• anyNumberOfTimes
• times(n)
• times(n..m)

Return arguments given as a sequence will be returned in the sequence they are given for each subsequent call to the mock:

The mocking support is still a work in progress. Significant omissions include friendly mock names, a more unified creation syntax, and more complex argument matchers (for example, Regex support: user should receive name(#/Guybr/)).

Design goals

I had basically two goals in mind for the syntax of mock creation:

• I wanted something that played nicely and looked good with ISpec, and in this respect I drew some familiar syntax constructs from RSpec mocking.
• I wanted a great deal of flexibility. In this I drew far more from Mocha than I did for RSpec, whose mocking framework I find continually limiting.

But in particular, I wanted to use the power of Ioke’s macros to allow me to specify complex mocks in a more concise way. In Mocha, being a Ruby library, you can specify your mocks as simple key-value pairs in a hash, but if you want to build a more complex mock you must break off into a new line and start constructing it using separate calls. Ioke allows me to chain multiple expectations implicitly against the same should receive call, and even to apply the same expectation to an entire group of them:

Next steps

I haven’t used the macro capabilities to their full power, and it shows in the code: much of it could reasonably be made much, much tighter through the use of macro syntax rather than simply leveraging the macro construct to manipulate conventional data structures. In particular the lack of that syntax manifests itself in a dismaying lack of consistency between different syntaxes for constructing mocks and stubs; unifying that is the number one thing I want to improve, structurally, in the framework.

But, most importantly, I haven’t yet used the framework in anger for anything; simply building it has consumed most of my Ioke hacking time. I suspect that it would benefit a great deal, as everything does, from real-world use; although I am an experienced tester with mocks, I am not yet an experienced user of my own. If you’re embarking on an Ioke project and need to test with mocks, please do let me know.

Tomorrow I’ll be hopping on a plane to Las Vegas for RailsConf ‘09, which comes at an interesting time for the community. The sagging economy, rise of alternative Ruby-like dynamic languages, and recent furor about sexism (about which I and many of my colleagues are pretty upset) make for a challenging set of obstacles for the technology that we all love. I’m curious about what the general mood will be, and I’m sure I’m not the only one.

I don’t know about other companies, but I know that ThoughtWorks has scaled back our involvement in RailsConf rather considerably this year, not because we don’t still like Rails but because sending people to Vegas in the middle of the week at the expense of billable or otherwise productive work is a luxury that all too few companies can afford these days. I’m glad I get to go and hope to repay our investment by blogging frequently about the sessions and sharing my thoughts on the goings-on. If there’s anything in particular you’d like to hear about, drop me a line at btguthrie@gmail.com and let me know. You can also follow me on Twitter.

In particular, if you’re interested in hacking on Ioke while you’re there please do let me know. I think it’s too late to suggest a BOFS session, but I may try to put something together anyway.

As for the recent controversy, my own feelings are perhaps best captured by Martin Fowler’s commentary, who reminds us yet again why we value his advice so much. Liz Keogh’s post is also fascinating. I could go on with more, but so many others have already said it better than I can; suffice it to say that I’m disappointed, and admire the creation but maintain my doubts about the creator.

In Ioke, a macro is a method that receives its arguments unevaluated. Instead of receiving a series of arguments, it receives a message. All code in Ioke is represented by message chains. A message looks something like this:

Take a look at that call to each. It receives two arguments, separated by a comma: the name of the variable to bind the incoming variable to, and the code for the loop. It’s equivalent to the following code in Ruby:

It’s pretty similar, except that in Ruby a block represents a special syntax form. In Ioke, calls to each follow the same syntax rules as any other kind of call, except that each is written as a macro, not a method. In the case above we don’t want to evaluate that first argument (name), because it doesn’t actually represent anything in the local scope. If you tried to evaluate it, you’d receive an error:

Macros allow a programmer to pick apart the arguments they receive and choose to evaluate them or not evaluate them, or evaluate them in particular contexts, as they so choose. They’re enormously powerful, but can be tricky to use, because without a structured argument list you’re never quite sure what you’re getting without taking a close look at the incoming message chain.

That’s where destructuring macros come in. dmacros are special kinds of meta-macros that allow you to write your code based on what you’re expecting from an incoming message. Here’s an example of a simple dmacro used by Message to allow you to turn transform any arbitrary code you give it into a message:

Think of them as a bit like multimethods: they perform the hard work of macro argument matching and allow you to write code that’s more clear and structured than it would otherwise be. Note that you still get access to the low-level call information that a regular macro would receive, so you can perform special tricks with the incoming arguments or the rest of the message chain if you’d like.

I’ve added an HTML formatter for Ioke’s testing framework, ISpec, and added TextMate bundle commands for test runners that use that formatter. I’ve shamelessly ripped off RSpec’s HTML formatter for a similar, though much less polished, style, and the results look something like this:

Because if you’re testing comprehensively, in my experience, you almost never need them. Unit and functional tests aren’t a panacea, but failures around incorrect types represent a miniscule proportion of the errors in the Ruby codebases I’ve worked on. You get them every once in a blue moon, but they are fixed easily and quickly, and everyone moves on with their lives. People still ask me sometimes if I miss static types, and my answer is always an unequivocal no. They eliminate a particular type of error that I almost never need to worry about.

Ola is also right that types aren’t the same as classes, and attempting to conflate the two will in all likelihood drive you towards class explosion and, ultimately, madness and penury.

That isn’t to say that interfaces shouldn’t have at least implicit contracts. In my own design, I try to follow the robustness principle whenever possible, and it improves code maintainability dramatically (nor does it does get discussed as a design principle nearly often enough). If it’s generally clear from reading the code what you can expect back from a method call and your types are suitably polymorphic then explicit type checks should be pretty rare.

The aim of software is, in a sense, to create an alternative reality. After all, when you use your cell phone, you simply want to push the fewest buttons possible and call, text, purchase, listen, download, e-mail, or browse. The power we all hold in our hands is shocking, yet it’s controlled by a few swipes of a finger. The drive to simplify the user’s contact with the machine has an inherent side effect of disguising the complexity of a given task. Over time, the users of any software are inured to the intricate nature of what they are doing. Also, as the software does more of the “thinking,” the user does less.

From My Manhattan Project: How I helped build the bomb that blew up Wall Street. Highly recommended; the article touches on the ethics of software development, user interface design, complexity, and the classic question of the degree to which a creator may be held responsible for the acts of his or her creation.

It is not Osinski’s fault that his software makes it very easy to perform some otherwise rather difficult algorithms; that is, after all, much of what a computer does. But it does make me wonder that he agreed to work on it understanding full well that the technique was built on entire mansions of cards, and it serves as an ample reminder that our moral compasses are so often attuned only to the values of those around us.

But what, then, is appropriate user design? Should business software be ugly, difficult to use? Should it, like a disassembled watch, expose its innards for all the world to poke and prod at? Must an operation that you suspect your user does not rightly understand always come with a warning label: “Don’t touch unless you really, truly know what you’re doing”? Isn’t that just condescending to your users?

Like everyone else, I’ve used my share of software that I absolutely loathed; software that made simple tasks hard and hard tasks devilishly impossible, their user experience an afterthought, a victim of shrinking budgets and small visions. If there’s anything I’d like to see improved in what you might term the ecosystem of enterprise software, it’s the fact that so much of it is, from a human point of view, really wretched. If you must make people use the stuff, don’t make them dread the experience.

But am I wrong? Should an ugly veneer serve as a reflection of inner ugliness, or of my own lack of understanding? I don’t know how many kinds of Photoshop filters work; should I as their user ask the designers of Photoshop for a poor interface design, so as to force me to pay a psychic penalty for my ignorance?

Of course not. As developers, our eagerness to please our clients and stakeholders is, after all, only human, and anyway, software projects do not have a great track record of success when it comes to satisfying the demands of their procurers. Some projects are wise and some are not; Osinki labored foolishly, although surely if he had not taken up the call some other equally talented developer would have. Agile development, I think, provides us with the tools to build software more rapidly and closer to spec than ever before; so let us then pick our projects wisely.