I happily announce the release of the first NeverBlock enabled activerecord adapter. The neverblock-postgresql-adapter. This is a beta release but I have been testing it for a while now with great results.
And while this is a big improvement it only requires you to replace the driver name in the connection to neverblock_postgresql instead of postgresql as described in the official neverblock blog
To make a long story short, this enables active record to issue queries in parallel, much like in a multi-threaded application. But this has several advantages over multi-threaded operations:
I have benchmarked this against the plain postgresql adapter using different workloads categorized as follows
Very Light : A single count statement
Light : A single count and a create
Moderate : 2 counts and a create wrapped in a transaction that rolls back
Heavy : 3 counts, a create and an update wrapped in a transaction that commits
Very Heavy : 3 counts, one conditional count (on a non-indexed field), a create and two updates all wrapped in a transaction that commits
(if you are wondering why these queries in particular, they were extracted from some other code)
All were issued 1000 times
The results came as follows:
As you can see, NeverBlock::AR is persistently faster than vanilla AR. It appears that such work loads generate linear increase for both AR and NeverBlock::AR as the NeverBlock advantage was almost the same
Another benchmark was performed to test the effect of increasing the connection count for NeverBlock::AR. We tested with 2, 4, 8, 16 and 32 connections.
The benchmark consisted of first running "select 1" 5000 times and then running"select sleep(10)" "select sleep(1)" 20 times for each configuration.
As you can probably guess, increasing connection count has very little effect if the queries are all very fast (you cannot beat "select 1") but if the queries are all slow, you will be able to double the performance by simply doubling the connection count.
I hope this gives you a glimpse of what's coming next. Watch this space
And while this is a big improvement it only requires you to replace the driver name in the connection to neverblock_postgresql instead of postgresql as described in the official neverblock blog
To make a long story short, this enables active record to issue queries in parallel, much like in a multi-threaded application. But this has several advantages over multi-threaded operations:
- Fibers are cheaper than threads so this solution is theoretically faster.
- NeverBlock does not require full thread safety, just avoid using globals and static variables for transient state.
- It integrates nicely in evented programs thus eliminating the performance drop which occurs with the introduction of threads in such environments
I have benchmarked this against the plain postgresql adapter using different workloads categorized as follows
Very Light : A single count statement
Light : A single count and a create
Moderate : 2 counts and a create wrapped in a transaction that rolls back
Heavy : 3 counts, a create and an update wrapped in a transaction that commits
Very Heavy : 3 counts, one conditional count (on a non-indexed field), a create and two updates all wrapped in a transaction that commits
(if you are wondering why these queries in particular, they were extracted from some other code)
All were issued 1000 times
The results came as follows:
As you can see, NeverBlock::AR is persistently faster than vanilla AR. It appears that such work loads generate linear increase for both AR and NeverBlock::AR as the NeverBlock advantage was almost the same
Another benchmark was performed to test the effect of increasing the connection count for NeverBlock::AR. We tested with 2, 4, 8, 16 and 32 connections.
The benchmark consisted of first running "select 1" 5000 times and then running
As you can probably guess, increasing connection count has very little effect if the queries are all very fast (you cannot beat "select 1") but if the queries are all slow, you will be able to double the performance by simply doubling the connection count.
I hope this gives you a glimpse of what's coming next. Watch this space
so you can give simple instructions on how to setup this up? Can't wait to try.
This link references the gem you need to install and provides a code example. Basically you need to install the neverblock, neverblock-pg and the adapter gems from github. And you need the pg gem as well.
Once I upload things to RubyForge it will be a single gem install that gets you all this.
could you give instructions for setting this up on rails? Will this work with rails in its current form?
Dude, what's up with all the Islamic stuff all over your blog? I don't even see hardcore Christian technical bloggers pushing their religion so much.
"May Allah guide us to the straight path. Amen"
No thanks.
I don't get it. If your long running queries are done with sleep(10) how can that possibly take less than 10 seconds to execute?
@about_rails_support
It is just around the corner, a few more days and you should be able to use that with Rails
@too_many_islamic_stuff
I am not sure what do you mean by pushing? this happens to be my blog, I am not posting Islamic stuff in comments at your blog!
@sleep_10_runs_in_less_than_10
A typo, it should read: "select sleep(1)", fixed now, thanks
Was this test run against a multi-threaded "AR normal"? [and with .allow_concurreny set to true?]
Oh wait I just noticed that AR setup that way uses one connection per thread--boo.
So a perhaps more accurate test would be NeverBlock against an AR thread pool, since rails 2.2 will have database connection pools apparently.
Just thinking out loud.
Apparently there are some problems with AR multi-threaded, even then, so NeverBlock would win there, if it can overcome those bugs:
http://www.ruby-forum.com/topic/115430
Also an interesting study would be to use 'real' long db queries.
Thanks!
-=R
Can you comment on the use of NeverBlock with Rails 2.2? Does it make sense to use NeverBlock or MySQLPlus in 2.2? Is there any conflict?
Can you comment on the use of NeverBlock with Rails 2.2? Does it make sense to use NeverBlock or MySQLPlus in 2.2? Is there any conflict?