New!

No time to start an auction? Try our

Credit Card Processing Calculator

TransFS DevBlog / Article RSS Feed

Apr 06

Goodbye BackgroundRb, Hello Workling & Starling

I recently completely overhauled our background task processing for TransFS.com.  For the past 6 months, we’ve been running BackgroundRb… and I’ve been fairly satisfied with its performance.  There are a lot of BackgroundRb haters out there, which I gather comes from some vicious problems with its usage of Drb in an earlier version.  However, by the time I found and installed BackgroundRb… it had been rewritten from the ground up using Packet, and it was both stable and quite powerful.

I started to get anxious about the amount of RAM being taken up by my backgroundrb workers.

Nonetheless, as I’ve continued to create new background processing tasks… I started to get anxious about the amount of RAM being taken up by my backgroundrb workers.  In addition, the BackgroundRb “style” has always seemed rather awkward to me.  It offers some nice features, for sure, that are hard to find in a single background processing package elsewhere.  But, at the end of the day, I started to wish that we were using something that was both lighter and easier to code against.

Finally, over the weekend… the final straw: I need to set up a new process on our Joyent server, and I didn’t have the RAM available.  So, after reading a bunch of documentation and evaluating several solutions, it was “Goodbye BackgroundRb, Hello Workling & Starling”.

As anyone who has looked into background processing knows… there are a TON of solutions out there.  It seems that anyone who is anyone has written a background processing library.  The Ruby community is rich with alternatives, none of them perfect, and many of them with avid supporters.

Here are a few interesting links if you want to do your own research:

Starling is dead simple to install and setup… you can just grab the gem from the starling repository on github.  Similarly, workling has a repo on github (what decent project doesn’t, these days??).  However, I had some problems getting workling to run properly with the latest version of starling.  Without getting into the details, it seems that a change was made to workling in November which broke some minor stuff with Starling compatibility… which is odd.  Anyway, I’ve got my own fork here, which fixes these problems for us.  Unfortunately, I couldn’t figure out another way around this… so my fix is a hack… otherwise, I’d try to get it incorporated back into the main tree.

After switching to Workling & Starling, we were using about 300MB less RAM, for the same functionality.

Bottom line is this:  With about a days work, I was able to completely replace BackgroundRb for all of our tasks… and so far I am very pleased with the results.  First of all, BackgroundRb used a full Rails process for each worker.  For us, this meant 90Mb times 4, plus an extra 50MB or so for the backgroundrb client.  In the Workling/Starling world, we only have to run a single worker process… so that is a considerable RAM savings.

Here’s a graph of the RAM usage on our site before and after the changes were deployed:

RAM Utilization Before/After Workling Upgrade

RAM Utilization Before/After Workling Upgrade

I think the only thing that was missing, feature-wise, from Workling was the ability to set up cron-style scheduling of background tasks.  BackgroundRb had a nice feature of allowing you to configure tasks to run at specific times from the convenience of the backgroundrb.yml file.

However… this could just as easily be solved using cron!  The easiest thing would be to set up a rake task that gets run by cron whenever I want.  Unfortunately, the drawback to this is that we’d have momentary spikes in RAM usage as these background cron/rake tasks load the Rails stack into memory so that they can do their thing.

To get cron-style scheduled background tasks with Workling, I use a simple cron script that sends a memcache message to Starling.

I think I came up with a clever solution for this.  Instead of a rake task… we now use an extremely lightweight ruby script that uses the memcache gem to send a simple queue message to Starling, triggering the background tasks that we want to run.  This allows our cron task to trigger scheduled tasks, but without any overhead at all, since the Starling/Workling processes are already running with the Rails stack fully-loaded.

So, our dead-simple cron task looks like this:

#!/usr/bin/env ruby
require "rubygems"
require "memcache"
 
host = "localhost"
port = 55555
starling = MemCache.new "#{host}:#{port}"
 
case ARGV[0]
  when 'send_reminders'
    starling.set 'reminder_workers__send_reminders', {}
  when 'run_seo_queries'
    starling.set 'seo_workers__run_seo_queries', {}
end

And we simply run a cron task with the correct argument, on schedule:

working_cron.rb send_reminders

This seems to work great, and it is extremely easy to modify. Cool!

Overall, BackgroundRb served us well… and I hold no grudges against it. However, I’m impressed so far with the easy setup and friendly interface that Workling offers. Stay tuned for an update on how it performs over time!

Share and Enjoy:
  • Print this article!
  • Digg
  • del.icio.us
  • Facebook
  • Google Bookmarks
  • Reddit
  • Twitter

← Back to blog index

Leave a Comment