gears
within
gears

I gave a presentation on Handshake at this past Tuesday’s Boston RubyGroup and had a great time. There was a large crowd (although I suspect most were there to see Hackety Hack in action) and I got a number of excellent questions at the end. In particular people were curious about performance characteristics and whether disabling a contract system in production mode is a wise idea. I’ll address the question of mechanisms for selectively disabling and enabling Handshake in particular classes in a future post.

I tend to think that it’s fine as long as you’re aware of the situation and plan for it. A contract system will always impose a performance overhead, even with languages that support it natively, and you can anticipate its absence by using conventional exception mechanisms at sensitive spots, like the places where your code interacts with data from the outside world. But I can’t admit to much experience in that regard.

Nonetheless it’s probably worth knowing the answer to the question of exactly how much of a performance overhead Handshake imposes. I’ve run some tests and the results aren’t pretty.

Methodology

To perform the tests, I created two classes modeled on the BankAccount example I presented on Tuesday. The first checks the class and methods by enforcing contracts on it, much like the example, while the second checks it by raising conventional exceptions. Seeing them side-by-side gives a good feeling for the added expressiveness you get with contracts. I’ve included them both below. For this test I made a large number of deposits and withdrawals on three different objects: one checked with Handshake, the underlying object wrapped by Handshake’s proxy, and one checked conventionally,

Finally, I created a simple one-line class that does nothing except include Handshake so I could get a feeling for the performance penalty it imposes on object creation (class Foo; include Handshake; end). For this test I created a large number of convention Ruby objects, objects that include Handshake and define contracts on the constructor, and objects that merely include Handshake without adding any contracts.

Infinity = 1.0/0

class BankAccountHandshake
  include Handshake

  invariant("balance must always be positive") { @balance >= 0 }

  positive_n = 0..Infinity

  contract positive_n => self
  def initialize(balance)
    @balance = balance
  end

  contract positive_n => anything
  before do |amount|
    assert( (@balance - amount) >= 0, "Amount: #{amount} must be less than balance: #{@balance}")
  end
  def withdraw(amount)
    @balance -= amount
  end

  contract positive_n => anything
  def deposit(amount)
    @balance += amount
  end

end

class BankAccountChecked

  def initialize(balance)
    raise Handshake::ContractViolation, 
      "Given balance #{balance} must be greater than or equal to 0" unless balance >= 0
    @balance = balance
    raise Handshake::ContractViolation,
      "balance must always be positive" unless @balance >= 0
  end

  def withdraw(amount)
    raise Handshake::ContractViolation,
      "Given amount #{amount} must be greater than or equal to 0" unless amount >= 0
    raise Handshake::ContractViolation,
      "Amount: #{amount} must be less than balance: #{@balance}" unless ((@balance - amount) >= 0)
    @balance -= amount
    raise Handshake::ContractViolation,
      "balance must always be positive" unless @balance >= 0
  end

  def deposit(amount)
    raise Handshake::ContractViolation,
      "Given amount #{amount} must be greater than or equal to 0" unless amount >= 0
    @balance += amount
    raise Handshake::ContractViolation,
      "balance must always be positive" unless @balance >= 0
  end

end

Deposit and withdrawal

Benchmark.bmbm do |bm|
  [ :handshake_enforced, :handshake_ignored, :nohandshake_checks ].each do |name|
    account = tests[name].call
    bm.report(name.to_s + "_iteration_50_000") do
      50_000.times { account.deposit 100; account.withdraw 100 }
    end
  end
end

Updated: After running this test, I altered the Proxy class to lazily create named methods each time method_missing is called. The performance increase is considerable-around 33%-and I haven’t released the fix yet but I’m adding a line to the tests below to reflect the improved performance.

Object creation

Benchmark.bmbm do |bm|
  [ :handshake_enforced, :handshake_nochecks, :nohandshake_checks ].each do |sym|
    bm.report(sym.to_s) { 100_000.times( &tests[sym] ) }
  end
end

Analysis

It’s much slower than I feared, so slow that I wonder whether my benchmarking methodology is flawed. A walk through the relevant architecture might be in order in order to determine why exactly performance takes a hit, but feel free to skip.

Handshake uses the included module callback to alias the relevant class’s new method. It replaces the existing method with one that checks constructor contracts and class invariants, actually instantiating the object as necessary. Rather than returning the instantiated object, it returns a new proxy object that wraps the original object. The Handshake::Proxy class, through a combination of method_missing and forwarding, intercepts all method calls, performs a lookup on the relevant method, and executes any defined contracts as necessary.

Although the problem doesn’t seem purely algorithmic, the large disparity in object-creation performance between the Handshake class that performs contract checks and the Handshake class that doesn’t (the checked constructor is twice as slow) suggests that there are ways to optimize the checking mechanism.

The other problem is that the system removes virtually every Ruby runtime optimization and redefines several language-level mechanisms. Method arity isn’t preserved (because it can’t be). Methods defined on Object are redefined in Proxy. Method_missing imposes its own performance penalty.

As I see it, then, this leaves two opportunities for further performance improvements: caching method calls and rewriting some portion of the library in C. I’ve played with the Ruby/C bridge but I’m expert in neither C nor the Ruby runtime. I’m also leery of these types of low-level solutions because providing Windows support is difficult, and because they wouldn’t be compatible with JRuby. I’m much more adept with Java than with C, so perhaps I’ll perform similar tests with the JRuby runtime and use that as a starting point. I’ll report back here if I get that far.