The Control Panel that you may be familiar with uses Delayed Job. This is a Rails-specific gem that uses the database as a queue, with a nicely packaged worker process that handles messages as they arrive. Because the Control Panel only ever talks to Rails from Rails, this worked extremely well.

However, our other systems were not homogenous – there are a number of different interfaces that needed to be instructed at various times and across various machines, and Delayed Job didn’t really fit the bill. In particular, there were some tasks that could only happen on certain servers – while Delayed Job let us have multiple worker processes on different boxes, it essentially managed a single queue, so it could not differentiate between messages for one worker and messages for another.

AMQP to the rescue

Because of this, we looked elsewhere for our central work queue. We decided to use the AMQP protocol, with RabbitMQ as our implementation. AMQP is a protocol for taking incoming messages, placing them on one of many queues and routing them to a destination reliably; RabbitMQ is a server, built in Erlang, that is pretty lightweight and has some nice clustering features built-in.

Into the Warren

When it came to getting our Rails applications to talk to RabbitMQ, we had a couple of choices.

There is an AMQP gem which looked great. Unfortunately, as it uses EventMachine internally, it caused a few problems with our Passenger-based deployments – EventMachine opens a background thread which can cause issues to the web-server controlled Passenger processes.

There was also a gem called bunny. This works synchronously, thus avoiding the EventMachine issues – you connect and bunny either says “here’s the next message” or “the queue is empty” and then you disconnect.

It looked like bunny was perfect for our Passenger-based apps, with the AMQP gem being used in the non-Passenger side of things (actually in the end, we went with bunny on both sides, but for other reasons). But there was still a couple of things that we wanted to do with the messages.

Firstly, we are Rubyists and AMQP (the protocol) is language agnostic. In particular, we wanted to put a hash of data onto the queue and have a hash of data presented to us at the other end. Secondly, security is important to us – especially when it comes to tasks like rebooting or reconfiguring a server – so we wanted to ensure that the contents of the message were not tampered with in transit.

For these reasons, we built Warren. This adds filters over the AQMP layer. In our case, we had one filter that automatically marshalls the data to and from YAML as it is put onto and taken off the queue and a second HMAC filter that, with one line of configuration, ensures that the data was not altered on its journey.

Warren itself has an adapter layer, so it can quite happily use either the AMQP gem or bunny to actually talk to the queue. Adding new adapters, or filters, is as simple as creating a new subclass – Warren detects it and adds it to its stack.

Bigwig

With Warren it is pretty simple to put a message on to the queue:

Warren::Queue.publish(@queue_name, @hash_of_data)

It is just as easy to pull the messages off the other end:

Warren::Queue.subscribe(@queue_name) do | hash_of_data |
  do_something_with hash_of_data
end

But how do we implement this mysterious do_something_with(hash_of_data) call?

For our purposes, the endpoint needs to do different things depending upon which machine we are on and which queue we are listening to. The key one is our “builder” system which handles reboots, migrations and deployments, but there are several others, which while not yet live, will be coming onstream soon.

So effectively, we needed a set of daemon processes that listen to given queues and handle the messages that they receive. This splits into two components – the daemon framework and the workers themselves – which lead to Bigwig (keeping the rabbit theme going, Bigwig was a character in Watership Down).

Bigwig expects a configuration file, telling it how to connect to the AMQP server and a folder full of plugins. Each plugin is a simple Ruby class that performs a given action. Bigwig takes the incoming messages, looks for a parameter within the Hash called “method” and then uses that to find a plugin. The plugin is then invoked and can perform whatever tasks it needs to, using any data from the incoming message.

For monitoring purposes, Bigwig automatically deals with any incoming messages with a method of “ping” – these simply write a message to the log file. We can then send ping messages to the queue at set intervals and monitor that the log file is updated when expected – if we don’t see the update then we trigger an alert, in case the queue, or Bigwig, is down.

If the queue itself goes down, for whatever reason, then Bigwig uses an intelligent reconnect pattern. At first, it retries often, to minimise the downtime, but if the queue stays down, it reconnects less frequently, giving the queue a chance to restart before being swamped with connections. Of course, if the queue fails, then the ping messages won’t make it through and our monitor should alert us.

Calling back

Our primary internal application creates a Task object, with a unique identifier when dispatching tasks across the queue. This ID is then passed onto the queue and each Bigwig plugin is told of it when it is invoked. The application itself makes the Tasks available over ActiveResource.

This means that the Bigwig plugins themselves don’t actually need data passing over the queue. Instead, when they are invoked, they call back into the primary application, using the Task ID as the key to their ActiveResource call. They can then extract the data they need from the Task object and use that to do whatever they need to do. When completed, they call the complete! method on the ActiveResource object and our application knows that the job has been done (alternatively, they can call error! and our app knows that something went wrong). All of these appear in an event feed on the application’s homepage – so we know at a glance what activity is happening across our systems and, more importantly, what has failed.

We could have implemented this using AMQP again – a ‘responses’ queue that our primary application listens to, with each worker putting a message back when it was done. But we chose to use ActiveResource, partly because it is so simple (Rails makes it super-easy to expose your objects as XML and ActiveResource makes it equally easy for other Ruby programs to call back to them), and partly for philosophical reasons. When the primary application places a message onto the queue, it doesn’t necessarily know how it will be handled. AMQP dispatches it and a listener deals with it – in our case the listener is likely to be a Bigwig instance, but doesn’t have to be. But when the callback is made, it is always made into our application, which is already geared up for completing tasks or marking them as having an error. As we know that it is a Rails app at the other end, we chose to follow the path of least-resistance and use the “Rails way”.

However, we did have one issue with ActiveResource. Most of our plugins actually shell out to invoke Ruby scripts, with the plugin unpacking the information from the Task and generating command line arguments. This is a slight security risk, just because it is feasible that someone could inject a malicious command into the command line parameters (although as this is all internal it should never be an issue and Ruby’s system command does escape any supplied parameters). So we are switching the scripts to accept environment variables, thus bypassing the shell and eliminating a whole category of potential attacks. Secondly, using ActiveResource in a non-Rails application caused some weird errors – but only when you tried to POST back to the ActiveResource object. It turns out that ActiveSupport was causing some shifting to occur in the background and that was knocking out some of our own code. The solution was simple – we had to ensure that the require 'activeresource' statement was the first thing in each script.

So that’s a quick peek into the workings of some of our internal systems. We put a lot of thought into how these different components should work together, as their reliability is fundamental to our success as a business. And while they are not infallible (nothing ever is) I think you will agree that they show that using Ruby and AMQP as the “glue” to tie different systems together is not only feasible, it’s actually quite easy.

Every Brightbox runs Ubuntu, which is an operating system built on top of the GPL Linux kernel. Our infrastructure is built upon Xen, Apache, Nginx, MySQL, Nagios and many other open source software projects; not least of which are Ruby and Rails themselves.

But whilst we benefit from this software, without contribution, free software is nothing, so we contribute anything we can. The most obvious of these are the Brightbox deployment gem and its associated server-side tools. These are extensions to Capistrano that help you get your application onto your Brightbox as quickly and easily as possible.

We also have our apt repository where we repackage a number of free software projects to make configuring your Brightbox as easy as possible. More details on the repository are available on the wiki, but the most notable are our Passenger and Ruby Enterprise Edition packages.

However, nowadays, the real place for sharing your code is on Github. We have a number of projects available there, ranging from the tiny to the large.

These include:

• Flashing rails

  A rails plugin that makes it simple to display flash messages in your views in a consistent manner.

• Rujitsu

  A simple gem that collects together a number of convenience methods and various helpers.

• RSpec-rails extensions

  A gem that tidies up specifying your code with RSpec-Rails.

• Object Factory

  Brightbox’s very own answer to Factory Girl or Machinist that lets you build your test data with minimal configuration and no fixtures.

• Altered Beast and Redmine.

  We have taken our own forks of two popular Rails applications. David’s version of Altered Beast handles the Brightbox forums and Redmine handles our internal bug tracking and task lists.

• Warren and Bigwig

  Last, but by no means least, we have Warren and Bigwig. These are our wrappers to AMQP and RabbitMQ.

  We use RabbitMQ internally to deliver messages across our various infrastructure systems and needed a simple way to interface our ruby code to Rabbit (which is implemented in Erlang).

  This led to Warren, our wrapper over the AMQP protocol that make it simple to post messages onto the queue.

  In order to receive and act on those messages, we also built Bigwig (no prizes for spotting the rabbit references there), which takes those messages and responds. Bigwig matches each incoming message against a set of plugins, each plugin being small and focused on a particular task. Unrecognised messages are discarded, ensuring that rogue commands can’t wreak havoc upon our network.
  UPDATE: It turns out that Bigwig isn’t quite ready yet, as a big chunk has been rewritten. We’ll get it out there as soon as we can.

As these are all free software projects, please take a look inside and poke around. Any suggestions, improvements, patches or forks will be gratefully received. Also, stay tuned for an announcement on a major project we are looking to start in the next couple of weeks.

Update 2: We’ve also put the code for Isitruby19.com onto Github, under an MIT licence. Please go to the forum if you have any questions.

However, there is a significant barrier to Ruby 1.9 adoption; the compatibility of the gems that we have all come to depend on.  

Which is why we’ve launched isitruby19.com - a site that tracks gems and lists whether they are 1.9 compatible.  Each time you come across a gem that works for you, drop by and leave a comment, so we all get a feel for which gems need some work and which are ready today.

We chose RSpec because of its philosophy of “getting the words right”; code is often easier to write than it is read. As these specifications are also our internal documentation they must be easy to read as well.

However, as some of this Behaviour-Driven and Story-Driven development is pretty new, there isn’t much guidance on best practice, especially when it comes to the “User Stories” (which form the basis of the system’s acceptance tests). With that in mind, we thought we’d share our basic process we follow for each new feature.

(Download the original presentation here)

By the way, there’s a very subtle bug in the code; no prizes if you spot it!

For those that don’t know, Rails generates an authentication token within your forms and verifies this token when the form is submitted back to your application. This prevents attackers from crafting malicious requests whilst pretending to be your authenticated user.

However, for certain types of request (supposedly those that cannot be generated from a browser) this authentication token is ignored – in order to make it simpler for automated API access to your application (using JSON, XML or a few other data transport types). Unfortunately, text/plain is wrongly included as one of these types.

Luckily, the fix is simple. The long-term solution is to upgrade your application to Rails 2.1.3 or 2.2.2 (when they are released); the quick fix is even easier – tell Rails to verify text/plain requests by creating a file (called mime_type_csrf_fix.rb) in your config/initializers folder:


# temporary fix for http://www.rorsecurity.info/journal/2008/11/19/circumvent-rails-csrf-protection.html
Mime::Type.unverifiable_types.delete(:text)


Because we use Ubuntu Dapper, which ships with Ruby 1.8.4, it includes an earlier version of the REXML library (for XML processing). As you may know, there was a recent security vulnerability to do with the REXML library and these latest two versions of Rails include a fix for this.

Unfortunately, the library that ships with 1.8.4 is slightly incorrect. Ruby conventions state that a constant should be all capitals so, quite rightly, the Rails updates check for a constant called REXML::VERSION. However, in 1.8.4, the constant is actually called REXML::Version meaning that Rails falls over with an “uninitialized constant REXML::VERSION” error.

The quick fix is to manually edit /usr/lib/ruby/1.8/rexml/rexml.rb and add a new (correctly named constant). You will need to use sudo to edit the file, as it is owned by root; after editing the file should look something like:

module REXML
        Copyright = "Copyright © 2001, 2002, 2003, 2004 Sean Russell <ser@germane-software.com>"
        Date = "2005/224"
        Version = "3.1.3"
        VERSION = "3.1.3"
end

We are also looking at our options for a permanent fix for this issue.

The time: Tuesday the 2nd September 2008, 9am CET.
The place: The Maritim proArte Hotel, Friedrichstrasse, Berlin.
The plan: Brightbox set up their booth and then have a leisurely stroll around Berlin, taking in the sights, sounds and Bratwurst, before a good night’s kip and a fantastic exhibition at RailsConf Europe.

That was the plan. Unfortunately KLM conspired against us. A two and a half hour journey rapidly expanded into a twenty three hour marathon, involving unidentifiable hotel food, a detour via Paris, lost luggage, hysterical, smelly geeks, truffle cake and “luxury” mojitos. We were also without Neil, who bravely volunteered to stay in England installing a load of new hardware in our new racks.

RailsConf Europe is pretty much the biggest Ruby on Rails event on our continent. We were told there were about 800 attendees here (although RailsConf US in Portland, earlier in the year, had twice as many). With that in mind Brightbox stepped up to become gold sponsors of the event, providing us with a booth between ELC Technologies, a global agile development shop, and Five Runs, the Rails profiling tool, who we know quite well.

We didn’t get the chance to see many sessions, but there was a pre-conference Q&A session on Tuesday evening with David Heinemeier Hansson (DHH), Jeremy Kemper and Michael Koziarski. It was interesting to hear DHH praising 37signals’ Xen virtualisation setup (this being the platform we also currently use), later on I chatted with him and explained how we’re working hard to get a production-ready Rails stack included with Ubuntu; which will help when setting up your servers, even if you don’t choose Brightbox.

DHH also opened the conference proper on Wednesday morning with his Keynote on dealing with legacy code. This was interesting to me as, like most developers, I am often overwhelmed by the urge to rewrite code that I wrote a few months ago. This was followed, later in the day, by Jeremy Kemper talking about performance improvements – in particular the new features in Rails that make use of the facilities that HTTP provides.

Overall, however, the consensus seemed to be that the talks were solid but uninspiring (apart from our very own Rob Lee with a talk on semantic markup, dressed in a Brightbox t-shirt).

At the booth things were mental – especially in the first break. All the free Brightbox t-shirts vanished in less than ten minutes, as did the “I love Ruby” stickers. However, our giveaway was nowhere near as good as ELCs – they had free beer!

We did meet some of our existing customers (a few for the first time in real life) and a lot of prospects. Interestingly, at least from our point of view, we saw a lot of interest in our Managed Cluster services (where we build and maintain a high availability cluster of boxes for you).

An evening out with a load of our friends from Yorkshire (the aforementioned Rob and Deb, Louisa, Paul and Charmagne) ended my involvement with RailsConf (I had to fly back early because my babby was starting school) but John and Jeremy stayed on till the very end.

Overall, the travel was horrible, the food was great (I had the best burger I’ve ever eaten at a hotel just off Friedrichstrasse) and the conference was good. Berlin is a magnificent city and we had a fantastic time with some old friends and made some new great new ones. So that’s our story – how was it for you?