JSON Benchmarks in jRuby

I am in the process of switching a major application from MRI Ruby (specifically 1.8.7-p302) using many C extensions to jRuby (currently trying 1.5.3-master). In my application, performance is extremely important. It is so important in fact, that I will be writing about some of my experiences in troubleshooting the speed and getting those important milliseconds back. When I am trying to keep an entire transaction from start to finish under 40ms and just the decoding of a JSON object into a Ruby object in jRuby takes roughly 30ms using json_pure, we may have to explore other avenues.

Just to verify that JSON was indeed as slow as I thought it was, here is a quick benchmark of it:

require 'json/pure'
Benchmark.bmbm do |x|
  x.report("parse:") { JSON.parse(json) }
end

JSON Pure Results:

Rehearsal ------------------------------------------
parse:   0.255000   0.000000   0.255000 (  0.255000)
--------------------------------- total: 0.255000sec

             user     system      total        real
parse:   0.024000   0.000000   0.024000 (  0.024000)
parse:   0.017000   0.000000   0.017000 (  0.017000)
parse:   0.032000   0.000000   0.032000 (  0.032000)
parse:   0.042000   0.000000   0.042000 (  0.041000)

Rehearsal -------------------------------------------
encode:   0.036000   0.000000   0.036000 (  0.035000)
---------------------------------- total: 0.036000sec

              user     system      total        real
encode:   0.010000   0.000000   0.010000 (  0.010000)
encode:   0.007000   0.000000   0.007000 (  0.006000)
encode:   0.006000   0.000000   0.006000 (  0.006000)
encode:   0.017000   0.000000   0.017000 (  0.016000)

That wasn’t so great. We’re looking at about 30ms on average to decode and an additional about 10ms to re-encode. If we want to keep our total transactions to 40ms and we haven’t even done anything with our data, we’re in pretty bad shape. Let’s try the same thing again, only this time with json-jruby.

require 'json/pure'
Benchmark.bmbm do |x|
  x.report("parse:") { JSON.parse(json) }
end

JSON-jRuby Results:

Rehearsal ------------------------------------------
parse:   0.009000   0.000000   0.009000 (  0.008000)
--------------------------------- total: 0.009000sec

             user     system      total        real
parse:   0.003000   0.000000   0.003000 (  0.002000)
parse:   0.007000   0.000000   0.007000 (  0.003000)
parse:   0.003000   0.000000   0.003000 (  0.003000)
parse:   0.002000   0.000000   0.002000 (  0.002000)

Rehearsal -------------------------------------------
encode:   0.009000   0.000000   0.009000 (  0.009000)
---------------------------------- total: 0.009000sec

              user     system      total        real
encode:   0.001000   0.000000   0.001000 (  0.001000)
encode:   0.001000   0.000000   0.001000 (  0.001000)
encode:   0.001000   0.000000   0.001000 (  0.001000)
encode:   0.001000   0.000000   0.001000 (  0.001000)

Much better, right. We’re averaging 2.5ms to decode it and 1ms to encode it. That’s much more tolerable.

Just for poops and laughs, we should compare this to our MRI version with C bindings (the version we were using before), the super fast yajl-ruby gem.

require 'yajl'
parser = Yajl::Parser.new
Benchmark.bm do |x|                                                                                
  x.report("parse:") { parser.parse(json) }
end
parsed = parser.parse(json)                                                                        
Benchmark.bm do |x|
  x.report("encode:") { parsed.to_json }
end

And here are the MRI Yajl-ruby results:

Rehearsal ------------------------------------------
parse:   0.000000   0.000000   0.000000 (  0.000159)
--------------------------------- total: 0.000000sec

      user     system      total        real
parse:  0.000000   0.000000   0.000000 (  0.000101)
parse:  0.000000   0.000000   0.000000 (  0.000093)
parse:  0.000000   0.000000   0.000000 (  0.000098)
parse:  0.000000   0.000000   0.000000 (  0.000093)

      user     system      total        real
encode:  0.000000   0.000000   0.000000 (  0.000063)
encode:  0.000000   0.000000   0.000000 (  0.000063)
encode:  0.000000   0.000000   0.000000 (  0.000064)
encode:  0.000000   0.000000   0.000000 (  0.000065)

These benchmarks are by no means exact, but they do give you a feel for the fact that jRuby is slower than MRI with C extensions. The comparison is essentially 3.5ms round trip in jRuby and about 1.5ms in MRI. It might not seem like a lot, but when every millisecond counts, there is a lot that can happen in those extra 2 milliseconds that could be available every transaction.

Note: I reran the code by hand and formatted the results by hand, so the code won’t exactly produce the results above. These benchmarks were all done on my 2008 MacBook Pro with 4G of RAM and an Intel 2.53GHz Core 2 Duo.

UPDATE: It has been pointed out to me that I didn’t take advantage of jRuby’s hotspot feature (Thanks NickSeiger and Charles Remes). Hotspot basically kicks in after it figures out the best ways to optimize the code. So here are some new results and it’s safe to note that json-jruby is so fast, that the computer can’t time it.

jRuby json_pure:
Rehearsal ------------------------------------------
parse:   0.001000   0.000000   0.001000 (  0.001000)
--------------------------------- total: 0.001000sec

             user     system      total        real
parse:   0.001000   0.000000   0.001000 (  0.000000)
Rehearsal -------------------------------------------
encode:   0.001000   0.000000   0.001000 (  0.001000)
---------------------------------- total: 0.001000sec

              user     system      total        real
encode:   0.000000   0.000000   0.000000 (  0.000000)

jRuby json-jruby:
Rehearsal ------------------------------------------
parse:   0.000000   0.000000   0.000000 (  0.000000)
--------------------------------- total: 0.000000sec

             user     system      total        real
parse:   0.000000   0.000000   0.000000 (  0.000000)
Rehearsal -------------------------------------------
encode:   0.000000   0.000000   0.000000 (  0.000000)
---------------------------------- total: 0.000000sec

              user     system      total        real
encode:   0.000000   0.000000   0.000000 (  0.000000)
Posted in jRuby. Tags: , , . 7 Comments »
  • Pingback: Tweets that mention JSON Benchmarks in jRuby | Erics Tech Blog -- Topsy.com

  • Anonymous

    To get some comparable numbers, have you thought about doing N.times { } within the benchmark to get non-zero results?

  • Anonymous

    Agree with @rykov,

    Benchmarking shoudl be done 100 times at least. Measuring a single occurrence do not exercise these VMs that support JIT or other optimizations.

  • http://eric.lubow.org Eric Lubow

    I should clarify exactly what I did. I apologize for the confusion. I actually ran this doing 25_000.times {} and 100_000.times {}. The reason it came out so small (I believe) is that the JSON object I used was about 10k rather then n Megs that are usually used for tests.nnTo this end, I decided to try doing 100_000.times { total_time += Benchmark.realtime { JSON.parse(json) } }; avg = total_time / 100_000; puts “Average decode time”;nnThis way the optimizations could take place. I also did it where it wasn’t an average and I could watch the times decrease as the optimizer took effect. The other way I tried things was to do 100_000 iterations prior to the benchmark and then run the benchmark once. This way the optimizer had already kicked in as well. Both yielded the same results of 0.00000ms.

  • Max

    A better way might be instead of running a fixed number of iterations, run for a fixed length of time, i.e. figure out how many iterations you can get through in 10 seconds or something. A little more complicated to write, but probably a better benchmark.

  • Pingback: Delicious Bookmarks for November 12th from 02:29 to 02:59 « Lâmôlabs

  • http://twitter.com/cowtowncoder Tatu Saloranta

    On jRuby, another highly performant approach could be using Jackson (http://jackson.codehaus.org) — I’m pretty sure it is being used, and one link I found was this [https://github.com/guyboertje/jrjackson]