Benchmarking Dynamic Method Creation in Ruby
Posted by matijs 22/04/2011 at 09h30
Let’s look at dynamic method generation. I need it for GirFFI, and if you
do any kind of metaprogramming, you probably need it too. It was
already shown a long time ago that using string evaluation is preferable to using
define_method with a block.
That is, if you care at all about speed.
How preferable? About a factor of 1.8 on Ruby 1.8.7:
user system total real
regular: 0.270000 0.000000 0.270000 ( 0.267164)
string eval: 0.270000 0.000000 0.270000 ( 0.273170)
define_method: 0.490000 0.000000 0.490000 ( 0.493258)
These numbers are relatively easy to explain. The string evaluation
basically has the exact same result as regular method definition, because
you actually do the same thing: You define a method using def. With
define_method, you have a lot of overhead because a block is more than a
bunch of code. It’s actually a closure, and Ruby has to set up the closure’s
binding every time you call the method.
(Aside: There are of course other factors to consider. Using string
evaluation gives you much more power to build exactly the method you want,
while the fact that the blocks passed to define_method are closures
allows you to do things now ordinary method can do.)
On Ruby 1.9.2, the results are quite similar, although the difference is now only a factor of 1.5:
user system total real
regular: 0.140000 0.010000 0.150000 ( 0.142998)
string eval: 0.140000 0.000000 0.140000 ( 0.141439)
define_method: 0.210000 0.000000 0.210000 ( 0.211936)
Too bad. It seems we’re stuck with string eval. Let’s look at JRuby:
user system total real
regular: 0.324000 0.000000 0.324000 ( 0.324000)
string eval: 0.208000 0.000000 0.208000 ( 0.208000)
define_method: 0.690000 0.000000 0.690000 ( 0.690000)
Wait, that can’t be right. Let’s run that again:
user system total real
regular: 0.424000 0.000000 0.424000 ( 0.424000)
string eval: 0.241000 0.000000 0.241000 ( 0.241000)
define_method: 0.756000 0.000000 0.756000 ( 0.756000)
Hm, it got a little slower, but the pattern is the same: The method defined using string eval is the fastest of the lot. What is going on here?
Quick, let’s try rubinius.
user system total real
regular: 0.348022 0.000000 0.348022 ( 0.183686)
string eval: 0.380024 0.004000 0.384024 ( 0.196908)
define_method: 0.412026 0.008001 0.420027 ( 0.215781)
Uh-huh. Again please.
user system total real
regular: 0.376023 0.004000 0.380023 ( 0.192683)
string eval: 0.356022 0.000000 0.356022 ( 0.189258)
define_method: 0.164010 0.000000 0.164010 ( 0.138058)
Huh? Like, maybe the benchmark is wrong?
Okay, let’s try bmbm instead of bm. For rubinius:
Rehearsal --------------------------------------------------
regular: 0.332020 0.004001 0.336021 ( 0.172166)
string eval: 0.352022 0.004000 0.356022 ( 0.185557)
define_method: 0.192012 0.000000 0.192012 ( 0.152656)
----------------------------------------- total: 0.884055sec
user system total real
regular: 0.052003 0.000000 0.052003 ( 0.050464)
string eval: 0.052003 0.000000 0.052003 ( 0.050471)
define_method: 0.076005 0.000000 0.076005 ( 0.076912)
That looks more sane. Let’s try JRuby:
Rehearsal --------------------------------------------------
regular: 0.408000 0.000000 0.408000 ( 0.408000)
string eval: 0.196000 0.000000 0.196000 ( 0.196000)
define_method: 0.657000 0.000000 0.657000 ( 0.657000)
----------------------------------------- total: 1.261000sec
user system total real
regular: 0.095000 0.000000 0.095000 ( 0.096000)
string eval: 0.109000 0.000000 0.109000 ( 0.109000)
define_method: 0.416000 0.000000 0.416000 ( 0.416000)
Much better. Notice that the difference between string eval and define_method is a stunning factor of four!
Now go back to rubinius. Did you notice how fast it was? That’s stunning.
So what’s the difference there between define_method and string eval? Not
so big. But the numbers are small so there may be some influence from the
environment. Let’s look at how things scale: 10 million iterations:
Rehearsal --------------------------------------------------
regular: 1.240078 0.004000 1.244078 ( 1.034290)
string eval: 1.076067 0.000000 1.076067 ( 1.058813)
define_method: 0.848053 0.000000 0.848053 ( 0.838424)
----------------------------------------- total: 3.168198sec
user system total real
regular: 0.496031 0.000000 0.496031 ( 0.496565)
string eval: 0.528033 0.000000 0.528033 ( 0.500421)
define_method: 0.864054 0.000000 0.864054 ( 0.863875)
That’s a factor of about 1.6, somewhere between MRI 1.8.7 and 1.9.2.
Conclusions
The basic conclusion holds: String evaluation leads to faster methods. How much of a difference it makes depends on which Ruby you’re using, and will probably depend on the particular method you’re creating.
Benchmarking is tricky: If you don’t watch out, you could draw the wrong conclusions.
Finally, Rubinius is fast. Really fast. Wow.
The Code
Benchmarks were generated with the following program:
<typo:code lang=“ruby”> require ‘benchmark’
class Foo def regular; end
eval “def stringeval; end”
define_method(:block) {} end
foo = Foo.new
n = 10_000_000
Benchmark.bmbm do |x| x.report(“regular: “) { n.times { foo.regular } } x.report(“string eval: “) { n.times { foo.stringeval } } x.report(“define_method:”) { n.times { foo.block } } end </typo:code>