Gears Within Gears

I gave a presentation on Handshake at this past Tuesday’s Boston RubyGroup and had a great time. There was a large crowd (although I suspect most were there to see Hackety Hack in action) and I got a number of excellent questions at the end. In particular people were curious about performance characteristics and whether disabling a contract system in production mode is a wise idea. I’ll address the question of mechanisms for selectively disabling and enabling Handshake in particular classes in a future post.

I tend to think that it’s fine as long as you’re aware of the situation and plan for it. A contract system will always impose a performance overhead, even with languages that support it natively, and you can anticipate its absence by using conventional exception mechanisms at sensitive spots, like the places where your code interacts with data from the outside world. But I can’t admit to much experience in that regard.

Nonetheless it’s probably worth knowing the answer to the question of exactly how much of a performance overhead Handshake imposes. I’ve run some tests and the results aren’t pretty.

Methodology

To perform the tests, I created two classes modeled on the BankAccount example I presented on Tuesday. The first checks the class and methods by enforcing contracts on it, much like the example, while the second checks it by raising conventional exceptions. Seeing them side-by-side gives a good feeling for the added expressiveness you get with contracts. I’ve included them both below. For this test I made a large number of deposits and withdrawals on three different objects: one checked with Handshake, the underlying object wrapped by Handshake’s proxy, and one checked conventionally,

Finally, I created a simple one-line class that does nothing except include Handshake so I could get a feeling for the performance penalty it imposes on object creation (class Foo; include Handshake; end). For this test I created a large number of convention Ruby objects, objects that include Handshake and define contracts on the constructor, and objects that merely include Handshake without adding any contracts.

Infinity = 1.0/0

class BankAccountHandshake
  include Handshake

  invariant("balance must always be positive") { @balance >= 0 }

  positive_n = 0..Infinity

  contract positive_n => self
  def initialize(balance)
    @balance = balance
  end

  contract positive_n => anything
  before do |amount|
    assert( (@balance - amount) >= 0, "Amount: #{amount} must be less than balance: #{@balance}")
  end
  def withdraw(amount)
    @balance -= amount
  end

  contract positive_n => anything
  def deposit(amount)
    @balance += amount
  end

end

class BankAccountChecked

  def initialize(balance)
    raise Handshake::ContractViolation, 
      "Given balance #{balance} must be greater than or equal to 0" unless balance >= 0
    @balance = balance
    raise Handshake::ContractViolation,
      "balance must always be positive" unless @balance >= 0
  end

  def withdraw(amount)
    raise Handshake::ContractViolation,
      "Given amount #{amount} must be greater than or equal to 0" unless amount >= 0
    raise Handshake::ContractViolation,
      "Amount: #{amount} must be less than balance: #{@balance}" unless ((@balance - amount) >= 0)
    @balance -= amount
    raise Handshake::ContractViolation,
      "balance must always be positive" unless @balance >= 0
  end

  def deposit(amount)
    raise Handshake::ContractViolation,
      "Given amount #{amount} must be greater than or equal to 0" unless amount >= 0
    @balance += amount
    raise Handshake::ContractViolation,
      "balance must always be positive" unless @balance >= 0
  end

end

Deposit and withdrawal

Benchmark.bmbm do |bm|
  [ :handshake_enforced, :handshake_ignored, :nohandshake_checks ].each do |name|
    account = tests[name].call
    bm.report(name.to_s + "_iteration_50_000") do
      50_000.times { account.deposit 100; account.withdraw 100 }
    end
  end
end

Updated: After running this test, I altered the Proxy class to lazily create named methods each time method_missing is called. The performance increase is considerable-around 33%-and I haven’t released the fix yet but I’m adding a line to the tests below to reflect the improved performance.

Object creation

Benchmark.bmbm do |bm|
  [ :handshake_enforced, :handshake_nochecks, :nohandshake_checks ].each do |sym|
    bm.report(sym.to_s) { 100_000.times( &tests[sym] ) }
  end
end

Analysis

It’s much slower than I feared, so slow that I wonder whether my benchmarking methodology is flawed. A walk through the relevant architecture might be in order in order to determine why exactly performance takes a hit, but feel free to skip.

Handshake uses the included module callback to alias the relevant class’s new method. It replaces the existing method with one that checks constructor contracts and class invariants, actually instantiating the object as necessary. Rather than returning the instantiated object, it returns a new proxy object that wraps the original object. The Handshake::Proxy class, through a combination of method_missing and forwarding, intercepts all method calls, performs a lookup on the relevant method, and executes any defined contracts as necessary.

Although the problem doesn’t seem purely algorithmic, the large disparity in object-creation performance between the Handshake class that performs contract checks and the Handshake class that doesn’t (the checked constructor is twice as slow) suggests that there are ways to optimize the checking mechanism.

The other problem is that the system removes virtually every Ruby runtime optimization and redefines several language-level mechanisms. Method arity isn’t preserved (because it can’t be). Methods defined on Object are redefined in Proxy. Method_missing imposes its own performance penalty.

As I see it, then, this leaves two opportunities for further performance improvements: caching method calls and rewriting some portion of the library in C. I’ve played with the Ruby/C bridge but I’m expert in neither C nor the Ruby runtime. I’m also leery of these types of low-level solutions because providing Windows support is difficult, and because they wouldn’t be compatible with JRuby. I’m much more adept with Java than with C, so perhaps I’ll perform similar tests with the JRuby runtime and use that as a starting point. I’ll report back here if I get that far.

Update: I’m not sure if I made this clear below, so just for the record: ThoughtWorks is a cool company with a great reputation and I’m thrilled to be joining them. All of my comments below should be taken in the spirit of that statement.

I’m tickled pink to announce that I’ve accepted a job with ThoughtWorks as some kind of Ruby guy and so I’m moving from Boston to Chicago to live the dream of 80-100% travel to places that are not Chicago. I thought it might be worth explaining briefly why I chose them.

In short: I get to work in Ruby (apparently), on a diverse range of projects, in diverse settings, at considerably greater job security than I might find at a startup; I expect to learn a tremendous amount about developing good software; it’s a change of scenery and a chance to travel; everyone at the interview impressed me, and were kind enough to let me geek out over Handshake, yea, even unto the whiteboard; and significantly, I couldn’t find anything credibly negative about the company, either from former employees or outside observers. If they suffer from a little hubris, well, there are far worse sins to have.

I’d also like to note that everyone I met at the interview event, in particular their founder Roy, was remarkably and even alarmingly candid. It was a breath of fresh air.

The job starts on June 18th, and shortly thereafter I’ll be attending ThoughtWorks University for six weeks. I’ve always wanted to see India. I also have to relocate to Chicago between now and then; I don’t know anything about the city, and if anyone has any advice about living and working there I’d love to hear it.

From the relevant email:

I’ve written such a document and attached it; ...

Thanks, Google.

Updated: Some email leaked through in the original image and I’ve fixed it. Remind me to be more zealous in my croppery.

But it had to be done. Handshake 0.3.0 (should it have been 0.2.2?) will not enforce any contracts unless the global $DEBUG flag is set (ruby -d). Should have gotten around to it a while ago. Luckily, it’s a very easy change: don’t alias :new unless $DEBUG. A proxy object is never created, therefore no contract barrier exists.

This is just a quick note that I’ll be giving a brief presentation on Handshake next Tuesday at the Boston Rubygroup. Once I finish the slides they’ll be another nice introduction. Unfortunately the other Tuesday presentation is on Hackety Hack, with which I cannot hope to compete. Such is life.

I’ve pushed out a new version of Handshake. The big-ticket item in 0.2 is that method argument contracts now support contracts on blocks. When a block is passed to a method that’s protected by a block contract, the block will be modified to check the arguments and return values for that block in much the same way that a normal contract is checked. Note that this will probably not work in Ruby 1.9, or at least will have to be modified for recursive contract checking.

Here’s the syntax:

class StringArray < Array
  contract :each, Block(String => anything) => self
end

Block checking in 0.2.0 involved reopening the Proc class; release 0.2.1 fixes that.

The new release also contains a new method, checked_self. Because of the way Handshake is implemented (with a proxy object), calls to private methods aren’t checked (because they’re called on the real object instead of the proxy object). checked_self returns the proxy object instance and calls made to that object will be contract checked.

I’ve also improved the documentation. You can find the rdoc at http://handshake.rubyforge.org, and the project page at http://rubyforge.org/projects/handshake/.

Thanks, The Economist:

When it comes to most dangerous products—be they drugs, cigarettes or fast cars—this newspaper advocates a more [classically] liberal approach than the American government does. But when it comes to handguns, automatic weapons and other things specifically designed to kill people, we believe control is necessary, not least because the failure to deal with such violent devices often means that other freedoms must be curtailed. Instead of a debate about guns, America is now having a debate about campus security.

I recommend that you all subscribe; it’s excellent. Much of their content is also available online for free. I promise I’ll return to technical stuff soon. link

I am not a constitutional scholar, but it’s my understanding that the Bill of Rights is silent on the issue of an inalienable right to one’s own talk show. Don Imus isn’t in jail. This is all about community speech and the government isn’t involved. This is precisely how it’s supposed to work.

I’m in Chicago tonight for another job interview tomorrow. Never been to Chicago before, so that’s very cool, and I’m excited about the interview. Not much more to say except that downtown Chicago is stunning (and I get a 28th-floor view here), and that overriding Proc#call does not in fact affect the behavior of yield. Nobody expects the Spanish inquisition:

class Proc
  def weird(*args)
    # Only get all weird if we ask for it.
    puts "weird call!" if args.length > 0 && args[0] == "weird"
    orig_call(*args)
  end
  alias :orig_call :call
  alias :call :weird
end

def weird_call(str, &block)
  block.call(str)
end

def weird_yield(str)
  yield(str)
end

and you get:

>> weird_call("weird") { true }
weird call!
=> true
>> weird_yield("weird") { true }
=> true

This is frustrating, because I wanted Handshake to be able to support:

  contract Block(String => Integer) => 1..5

For example. Am I wrong? Or is there another mechanism for overriding the behavior of yield?

The degree to which Ruby’s core methods don’t appear to be orthogonal, instead employing lower-level mechanisms to do their work (Class#=== vs. Object#is_a? is another example) has been an unpleasant surprise to me.

Bigelow Tea, a regular advertiser on Mr. Imus’s radio show, said it was suspending its current advertising and re-evaluating its future relationship with the show.

Apparently there exists some non-trivial subset of human beings who reside at the interstices of shock radio and soothing, understated beverages. Are they advertising herbal teas, I wonder, or caffeinated?

P&G, Others Pull Imus Ads (subscription required)