Gears Within Gears

I should have posted these a long time ago, but better late than never: the slides from the presentation I gave at last month’s Boston RubyGroup on Handshake are available here in PDF format, and I’ve embedded the video below. I hate seeing myself on camera, so I haven’t watched it yet. Stay put for the big finish, where my laptop’s battery runs out of juice.

I haven’t had much chance to work on Handshake recently, no excuse really, but with any luck I’ll get a chance to push out another release of that and RubySearch before Sunday. Then I’m on to Chicago for a two-day initiation at ThoughtWorks, my new employer, followed by a six-week training session in India. I can’t wait.

I gave a presentation on Handshake at this past Tuesday’s Boston RubyGroup and had a great time. There was a large crowd (although I suspect most were there to see Hackety Hack in action) and I got a number of excellent questions at the end. In particular people were curious about performance characteristics and whether disabling a contract system in production mode is a wise idea. I’ll address the question of mechanisms for selectively disabling and enabling Handshake in particular classes in a future post.

I tend to think that it’s fine as long as you’re aware of the situation and plan for it. A contract system will always impose a performance overhead, even with languages that support it natively, and you can anticipate its absence by using conventional exception mechanisms at sensitive spots, like the places where your code interacts with data from the outside world. But I can’t admit to much experience in that regard.

Nonetheless it’s probably worth knowing the answer to the question of exactly how much of a performance overhead Handshake imposes. I’ve run some tests and the results aren’t pretty.

Methodology

To perform the tests, I created two classes modeled on the BankAccount example I presented on Tuesday. The first checks the class and methods by enforcing contracts on it, much like the example, while the second checks it by raising conventional exceptions. Seeing them side-by-side gives a good feeling for the added expressiveness you get with contracts. I’ve included them both below. For this test I made a large number of deposits and withdrawals on three different objects: one checked with Handshake, the underlying object wrapped by Handshake’s proxy, and one checked conventionally,

Finally, I created a simple one-line class that does nothing except include Handshake so I could get a feeling for the performance penalty it imposes on object creation (class Foo; include Handshake; end). For this test I created a large number of convention Ruby objects, objects that include Handshake and define contracts on the constructor, and objects that merely include Handshake without adding any contracts.

Infinity = 1.0/0

class BankAccountHandshake
  include Handshake

  invariant("balance must always be positive") { @balance >= 0 }

  positive_n = 0..Infinity

  contract positive_n => self
  def initialize(balance)
    @balance = balance
  end

  contract positive_n => anything
  before do |amount|
    assert( (@balance - amount) >= 0, "Amount: #{amount} must be less than balance: #{@balance}")
  end
  def withdraw(amount)
    @balance -= amount
  end

  contract positive_n => anything
  def deposit(amount)
    @balance += amount
  end

end

class BankAccountChecked

  def initialize(balance)
    raise Handshake::ContractViolation, 
      "Given balance #{balance} must be greater than or equal to 0" unless balance >= 0
    @balance = balance
    raise Handshake::ContractViolation,
      "balance must always be positive" unless @balance >= 0
  end

  def withdraw(amount)
    raise Handshake::ContractViolation,
      "Given amount #{amount} must be greater than or equal to 0" unless amount >= 0
    raise Handshake::ContractViolation,
      "Amount: #{amount} must be less than balance: #{@balance}" unless ((@balance - amount) >= 0)
    @balance -= amount
    raise Handshake::ContractViolation,
      "balance must always be positive" unless @balance >= 0
  end

  def deposit(amount)
    raise Handshake::ContractViolation,
      "Given amount #{amount} must be greater than or equal to 0" unless amount >= 0
    @balance += amount
    raise Handshake::ContractViolation,
      "balance must always be positive" unless @balance >= 0
  end

end

Deposit and withdrawal

Benchmark.bmbm do |bm|
  [ :handshake_enforced, :handshake_ignored, :nohandshake_checks ].each do |name|
    account = tests[name].call
    bm.report(name.to_s + "_iteration_50_000") do
      50_000.times { account.deposit 100; account.withdraw 100 }
    end
  end
end

Updated: After running this test, I altered the Proxy class to lazily create named methods each time method_missing is called. The performance increase is considerable-around 33%-and I haven’t released the fix yet but I’m adding a line to the tests below to reflect the improved performance.

Object creation

Benchmark.bmbm do |bm|
  [ :handshake_enforced, :handshake_nochecks, :nohandshake_checks ].each do |sym|
    bm.report(sym.to_s) { 100_000.times( &tests[sym] ) }
  end
end

Analysis

It’s much slower than I feared, so slow that I wonder whether my benchmarking methodology is flawed. A walk through the relevant architecture might be in order in order to determine why exactly performance takes a hit, but feel free to skip.

Handshake uses the included module callback to alias the relevant class’s new method. It replaces the existing method with one that checks constructor contracts and class invariants, actually instantiating the object as necessary. Rather than returning the instantiated object, it returns a new proxy object that wraps the original object. The Handshake::Proxy class, through a combination of method_missing and forwarding, intercepts all method calls, performs a lookup on the relevant method, and executes any defined contracts as necessary.

Although the problem doesn’t seem purely algorithmic, the large disparity in object-creation performance between the Handshake class that performs contract checks and the Handshake class that doesn’t (the checked constructor is twice as slow) suggests that there are ways to optimize the checking mechanism.

The other problem is that the system removes virtually every Ruby runtime optimization and redefines several language-level mechanisms. Method arity isn’t preserved (because it can’t be). Methods defined on Object are redefined in Proxy. Method_missing imposes its own performance penalty.

As I see it, then, this leaves two opportunities for further performance improvements: caching method calls and rewriting some portion of the library in C. I’ve played with the Ruby/C bridge but I’m expert in neither C nor the Ruby runtime. I’m also leery of these types of low-level solutions because providing Windows support is difficult, and because they wouldn’t be compatible with JRuby. I’m much more adept with Java than with C, so perhaps I’ll perform similar tests with the JRuby runtime and use that as a starting point. I’ll report back here if I get that far.

But it had to be done. Handshake 0.3.0 (should it have been 0.2.2?) will not enforce any contracts unless the global $DEBUG flag is set (ruby -d). Should have gotten around to it a while ago. Luckily, it’s a very easy change: don’t alias :new unless $DEBUG. A proxy object is never created, therefore no contract barrier exists.

This is just a quick note that I’ll be giving a brief presentation on Handshake next Tuesday at the Boston Rubygroup. Once I finish the slides they’ll be another nice introduction. Unfortunately the other Tuesday presentation is on Hackety Hack, with which I cannot hope to compete. Such is life.

I’ve pushed out a new version of Handshake. The big-ticket item in 0.2 is that method argument contracts now support contracts on blocks. When a block is passed to a method that’s protected by a block contract, the block will be modified to check the arguments and return values for that block in much the same way that a normal contract is checked. Note that this will probably not work in Ruby 1.9, or at least will have to be modified for recursive contract checking.

Here’s the syntax:

class StringArray < Array
  contract :each, Block(String => anything) => self
end

Block checking in 0.2.0 involved reopening the Proc class; release 0.2.1 fixes that.

The new release also contains a new method, checked_self. Because of the way Handshake is implemented (with a proxy object), calls to private methods aren’t checked (because they’re called on the real object instead of the proxy object). checked_self returns the proxy object instance and calls made to that object will be contract checked.

I’ve also improved the documentation. You can find the rdoc at http://handshake.rubyforge.org, and the project page at http://rubyforge.org/projects/handshake/.

I’m in Chicago tonight for another job interview tomorrow. Never been to Chicago before, so that’s very cool, and I’m excited about the interview. Not much more to say except that downtown Chicago is stunning (and I get a 28th-floor view here), and that overriding Proc#call does not in fact affect the behavior of yield. Nobody expects the Spanish inquisition:

class Proc
  def weird(*args)
    # Only get all weird if we ask for it.
    puts "weird call!" if args.length > 0 && args[0] == "weird"
    orig_call(*args)
  end
  alias :orig_call :call
  alias :call :weird
end

def weird_call(str, &block)
  block.call(str)
end

def weird_yield(str)
  yield(str)
end

and you get:

>> weird_call("weird") { true }
weird call!
=> true
>> weird_yield("weird") { true }
=> true

This is frustrating, because I wanted Handshake to be able to support:

  contract Block(String => Integer) => 1..5

For example. Am I wrong? Or is there another mechanism for overriding the behavior of yield?

The degree to which Ruby’s core methods don’t appear to be orthogonal, instead employing lower-level mechanisms to do their work (Class#=== vs. Object#is_a? is another example) has been an unpleasant surprise to me.

The job search: It’s hard to tell, but I think it went well. When a blog post rhymes, I eat limes. Friday is here, and I’m drinking a beer! Let’s end this.

California was quite nice in the daytime. I flew the redeye back this morning and I’ve been napping all day as a result. JetBlue may have a lot of forward leg room, but from side-to-side it’s torture. I did meet a couple of cool people, though. To my right, this guy. To my left, director of development here. I enjoy a little schmoozing now and then.

Blog news: Upgraded to Typo 4.1. Pain in the butt. Broke some CSS. Backend seems nicer. We’ll see.

Other: I did manage to get a little bit of work done. Expect a new release of Handshake soon with some small updates (documentation, checked_self, no blame system yet).

I’ve released Handshake as a live RubyGem. After you update your gem index you should be able to get it with gem install handshake, and you can browse the RDoc at handshake.rubyforge.org. A couple of things have changed since my previous blog post, so pay attention:

• Pre- and post-conditions now expect you to check assertions rather than return a boolean value from the block. I changed this because the project I’m working on requires some moderately complex rule-checking and trying to string together &&’s got really ponderous. This is probably an indication that there are better conceptualizations of that particular problem. I do quite dislike invariant and before/after employing different usage patterns.
• I’ve added a few new clauses. Check the RDoc if you need more information on these.
• I’ve fixed a lot of bugs, specifically a couple dealing with class type checking and contract inheritance.
• Contract violation error messages have been improved. The stack traces are also slightly better but not yet up to par. To generate a proper stack trace it has to be seen as originating from the method in question rather than somewhere within the bowels of Handshake, and that’s been improved but in the worst case is still two or three levels removed from the caller.

I’ll push out another release once I have a better handle on error messages. I’m having a difficult time finding an authoritative answer to what the “best” way is of getting detailed stacktrace information. Is Binding.of_caller still broken? Will debug.rb get me what I need? Or call_stack?

ruby-contract’s default behavior is to ignore contract checking unless your application’s $DEBUG flag is set. Although I haven’t made that the default behavior in Handshake, it makes a lot of sense; a contract system would probably impose inexcusable performance penalties without some way of turning it off. And if we’re not worried about performance in the first place it seems to me like a pure-Ruby implementation of something like one of the above is preferable.

As an addendum, I should note that hoe is totally pimpin’. Hyuck hyuck.

Florian Groß (cool!) asks:

Interesting. Any chance of getting a comparison to ruby-contract? :)

ruby-contract is quite nice, and if I’d realized just how nice beforehand I probably wouldn’t have bothered with Handshake; imagine my chagrin. Its wiki is defunct, and I couldn’t gem install ruby-contract, so I figured the project was dead or unmaintained and never took a look. Lucky for me, there are enough big differences between the two in interface and approach that it’s worth spending some time to examine them. I can’t claim to understand everything that’s going on it it yet, so Florian, please correct me if I get it wrong.

Update: I misread one of Florian’s signature examples. Fixed.

Big Picture

See my previous post for the big picture on Handshake. The idea is that you define contracts directly in classes, whose behaviors are applied to subclasses. In contrast, ruby-contract’s contracts reside in their own class and are applied to other classes through the fulfills method. The code below comes from ruby-contract’s test cases (tc_contract.rb) and gives a good feel for how you would interact with the code to define contracts.

  class EnumerableContract < Contract
    provides :each
  end

  module List
    module NonEmpty; end
  end

  class ListContract < Contract
    implies List

    def self.check_count() 3 end

    provides :each
    provides :[]
    provides :size do
      assert(@object.size >= 0)
    end
  end

  class NonEmptyListContract < ListContract
    implies List::NonEmpty

    provides :size do
      assert(@object.size > 0)
    end
  end

  class EmptyList
    def each() end
    def [](idx) end
    def size() 0 end

    fulfills ListContract
  end

  class NonEmptyList < EmptyList
    def size() 1 end

    fulfills NonEmptyListContract
  end

This code does a fair number of things that Handshake doesn’t attempt. Because Handshake agreements are defined either directly within a class or within its superclass there’s no way or reason to verify that a particular class provides or doesn’t provide a method. I also really, really like the syntax for the provides precondition checks. On the other hand, ruby-contract doesn’t appear to allow you to check class invariants, or to check postconditions in the context of all of a method’s arguments. Florian, please correct me if I’m mistaken.

Method Signature Checking

Both libraries offer method signature checking. I happen to prefer my syntax, but it’s mostly an aesthetics thing. I’m dismayed to discover that my idea of using the === operator for these clauses isn’t half as clever as I thought it was, or perhaps we are simply both clever. :)

# accepts a string, return value doesn't matter
contract :name=, String => anything

# accepts a block, returns an array
contract :each, Block => Array

# Matches something that is an Enumerable and that responds to
# either :to_ary or :to_a.
contract :x, all?( 
  Enumerable, 
  any?( 
    responds_to?(:to_a), 
    responds_to?(:to_ary) ) ) => anything

# accepts a string, return value doesn't matter
signature(:name=, String)

# accepts a block, returns an array
signature(:each, :block => true, :result => Array)

# Matches something that is an Enumerable and that responds to
# either :to_ary or :to_a.
# From ruby-contract docs.
signature :x, Contract::Check::All[
  Enumerable,
  Contract::Check::Any[
    Contract::Check::Quack[:to_a],
    Contract::Check::Quack[:to_ary]
  ]
]

We both provided remarkably similar combinators for signature clauses, although mine are defined methods and his are classes. Mine are quite expressive in terms of the error messages they generate, but I haven’t checked ruby-contract in that regard.

Implementation

Handshake is implemented as a proxy object that resides on top of an object and intercepts all method calls, running them through invariant, signature, and general pre- and post-condition checks. It uses this to ascertain blame as necessary, although the exception messages are generated by the checks themselves. The proxy object looks, acts, and feels like the real object.

In contrast, if I understand things correctly, ruby-contract uses eval to redefine methods that the programmer wants to check. For end users, the difference lies in the way calls to methods within the same class are evaluated. In Handshake, the proxy object establishes a barrier but cannot monitor anything that occurs inside that barrier. This means that calls to private methods are effectively uncheckable. The upside is that it allows you to define invariants in terms of methods as well as instance variables without getting stuck in an infinite loop. This is particularly useful for cases where you want to extend the behavior of a core Ruby class but don’t have access to its internals; see the NonEmptyArray example from my last post.

ruby-contract also hooks into Test::Unit to allow access to its assert methods within provides blocks and utilizes its inheritance properties to automatically check those clauses as soon as a class is defined as fulfilling a contract. This is an insanely good idea and I may stealborrow some portion of it.

Conclusion

Both libraries provide some unique and complementary contract mechanisms. ruby-contract is more mature, judging by the code, although it hasn’t been updated for a while. Not to be all wishy-washy, but it looks like there are projects that could benefit from either library, or both. I’m glad I finally took the time to dig through it.

Handshake is an informal design by contract system written in pure Ruby. It’s designed to offer simple, flexible semantics for improving the clarity and robustness of your code. It supports three basic kinds of checking: invariants, method pre/post conditions, and method arguments. This is a unit-tests-pass-but-you-probably-shouldn’t-use-it-for-anything-important 0.1 release.

A couple of clarifications: Pre- and post-conditions, and argument contracts, all accept the name of the method you’d like them to apply to as an optional first argument. If you omit it, they’ll be applied to the next method you define.

Also, anything is a clause that always returns true.

Updated 5/9/07: I’ve modified some of the content below to reflect changes to Handshake’s interface. Handshake is also now obtainable as a gem (gem install handshake).

Argument Contracts

Argument contracts are insanely handy. They allow you to specify conditions for the arguments and return values of methods simply and easily. Here’s an example:

class AcceptsString
  include Handshake
  contract [ String, Fixnum ] => anything
  def initialize(str, num); ...; end
end

Wait a second, I hear you say. That’s not exciting at all! I just left Java behind forever. Quasi-statically-typed languages are for suckers.

Here’s the rub: Handshake accepts as an argument clause any object that implements the === method. Why? Because it’s an incredibly convenient “is equivalent to” form of comparison. It’s what makes Ruby’s case statement so handy. Try this awkwardly contrived example on for size:

# Rate a website on a scale from 1 to 5.
class WebsiteRatingSystem
  include Handshake

  contract [ String, /^http:\/\// ] => anything
  def initialize(name, url)
    @name, @url, @ratings = name, url, []
  end

  contract 1..5 => anything
  def add_rating(n); @ratings << n; end

  contract nil => 1..5
  def average_rating
    ratings_sum = @ratings.inject(0.0) { |sum, n| sum + n }
    ratings_sum / @ratings.length
  end

  contract [ String, [ Integer ], Block([String, Integer] => anything) ] => Hash
  def generate_report(name, *metrics, &block)
    ..
  end
end

But Brian, you say, in the imaginary conversation I am conducting with you in my head. Brian, I like being able to give it a range, or a regexp. That seems downright neato. But man, what a hassle to have to craft my own ===-implementing object for every single clause.

As it turns out, I have made this process entirely hassle-free for you.

contract assert("equals foo") { |arg| arg == "foo" } => anything
contract any?(String, Integer) => anything
contract [ all?( hash_of?(Symbol, String), 
                 hash_with_keys(:foo, :bar, :baz) ] => anything
contract nonzero? => hash_contract(:count => Integer, :description => String)
contract responds_to?(:each, :map, :length) => anything
contract nil => String

And they said that duck typing and contracts didn’t mix! Good day, sir and/or madam.

Except for also

I’ve defined some accessors, just for you.

class HoldsSomeThings
  include Handshake
  contract_accessor :foo => String, :bar => Integer
  contract_reader ...
  contract_writer ... # You get the idea.
end

Invariants

Also handy are invariants, which are conditions that should always hold. They’re checked after your object is constructed and before and after every method invocation. They have access to local methods and instance variables. They are all up in your class, policing your object.

class NonEmptyArray < Array
  include Handshake
  invariant { not empty? }
end

Pre- and post-conditions

Sometimes you need a broader, more holistic approach to method checking. These conditions, if specified, are executed before and after (respectively) the named method and have access to its arguments, return value(s), and any local methods and instance variables. These work very much like you’d expect them to:

class HoldOnJustAMinute
  include Handshake

  before("halt in the name of the law") do |arg|
    assert @justice_dispensable
    assert arg.law_abiding?
  end; def do_something_potentially_damaging(arg)
    ...
  end
end

Caveats

• It isn’t underpinned by a formal calculus (although it’s modeled very loosely on the PLT Scheme contract library and on the Eiffel contract system). I’m more or less cool with that; Ruby’s appeal to me comes not least from its loose informalism. But the professor under whose grudging tutelage I developed this would like you to know that until it does it’s not good for anything whatsoever, and whatever benefits you think you might be deriving from such a seemingly useful library are just a figment of your diseased imagination. He informs me that he’s okay with that bit, but the half-bakedness of the implementation bothers him more. See below.
• It doesn’t yet have a proper blame system. See abovebelow re: professor.
• It handles inheritance only trivially: a subclass will inherit the contracts of its superclass but will not prevent you from overriding them. This means that it allows you to write contracts that fail the Liskov substitution principle. I hope to fix this soon, but in the meantime, don’t let it keep you awake at night.
• There’s a lot of metaclass magic going on under the covers, and you shouldn’t use it in contexts with a lot of other metaclass magic (myrailsapp/lib is probably fine; myrailsapp/app is not.)
• There’s no rdoc yet (although the code itself is fairly well-documented). Fixed. See handshake.rubyforge.org.
• You can’t define contracts in modules.
• And it will probably impact performance, although I haven’t conducted any performance testing.

Update: The inimitable Matthias Felleisen responds via email:

1. I never insist on a formal calculus. That’s what researchers are for. It’s a tool for exploring the ideas, communicating them in concise form, and proving basic fundamental theorems about such frameworks.

2. I do think that half-baked or quarter-baked implementations of good ideas are bad for the construction of quality software in the long term. They just propagate the silly idea that programming is shallow and anyone can do it, trained or not. One day we’ll have the equivalent of bridges come down on us in software and then the regulators will be worse than anyone can imagine now.

3. I also believe that without a properly design blame system, a contract framework provides only a 1/3 of the value that it can. That’s sad. It’s like buying a model T when we know how to build Porsches.

In my defense, this is at least two-thirds more-baked than any other Ruby contract system (5/9/07: except flgr’s ruby-contract. But I still like mine better). Since no Ruby contract system that I’ve seen is more than 15% baked, I guess that would make this… yes, about a quarter baked. It’s also a lot of fun to work on.

Finally

Find the project online at rubyforge.org/projects/handshake (MIT license), the RDoc at handshake.rubyforge.org, and the gem handshake, and please do let me know if you end up using it, and how.