In the three and a half years since Brightbox started, we’ve built a solid reputation for our “Serious Rails Hosting” platform. We’ve spent the last twelve months building Brightbox Cloud our new cloud Infrastructure-as-a-Service which is already making big waves. We’re now looking for an experienced software engineer to join our small but highly productive development team. This is an awesome opportunity to use your skills and experience to help shape our future products and direction in this exciting and fast growing industry.

Main responsibilities:

• Lead development of our distributed cloud management and automation systems
• Break down complex projects into clear tasks and specs
• Proactively find ways of improving our services, systems and code
• Efficiently manage your own workload as well as that of others
• Extract code to be released as free software projects

The ideal candidate:

You have an insatiable desire to understand things, to break them apart and to fix them. You are continuously looking for ways to improve things. You are enthusiastic about some technologies and vocalise your reasoned dislike for others. You have strong opinions and stand by them when it really matters, but are willing to compromise when it doesn’t. You are able to communicate your ideas succinctly with eloquence and clarity. You have probably contributed to some free software projects, and perhaps started your own. You don’t take yourself too seriously and are comfortable with others reviewing your work. You love what you do.

Skills & Experience:

• Several years experience of coding in Ruby
• Experience with Ruby web frameworks, such as Rails, Sinatra
• Experience of testing frameworks, such as rspec
• Experience of deployment using tools, such as Capistrano
• In-depth knowledge of distributed cloud infrastructure concepts
• Experience programming asynchronous systems
• Experience consuming and creating APIs
• Experience of working in a fast-paced, agile environment
• Ability to empathise with users

Our team:

We’re Ruby developers and system engineers. We’re obsessive about great user experience and clear documentation.  We’re passionate about high availability, performance and consistency, though not all at once. We’ve coded seriously in Ruby, C, Delphi, Ada, C++, Python, Amiga AMOS, PHP, Java, Objective-C, Pascal, Eiffel, Basic, Perl, Informix 4GL, Javascript and Bash. We use Ubuntu, Centos and OS X. We take our work seriously but we have a lot of fun.

Like our systems, our teams are geographically distributed, communicating in real-time throughout the working day using our private Jabber conference or phone calls. We manage our work with Redmine, usually organising chunks of work into weekly sprints. We’re a lean organisation – you’ll have direct access to leadership as well as other teams. Our development, engineering and operations teams all work closely together which shortens the feedback loop and means we get things done better.

Location:

Remote, but ideally working within 2 hours of UTC.

Salary:

circa £55k p.a. (depending on experience).

Closing date:

Friday 22nd April 2011.

How to apply:

Send a hello email and a CV (PDF or plain text) to jobs@brightbox.co.uk. All correspondence from recruitment agents must be in Esperanto and to our special recruitment email address: root@localhost

This is generally considered to be best practice where ever a server is geographically located, however, it may  cause a problem if you are hosting a  geographically specific web application.

If your web application requires a different time-zone such as CET, BST or PST, or Daylight Saving Time is giving you a headache,  take a look at our Wiki page for a timely introduction to the subject of time-zones on Brightboxes.

Again, we’re sponsoring the event and have provided hosting for the website. Our technical director John Leach is also speaking at the conference – his talk “UNIX: Rediscovering the wheel” will take place in the Great Hall on Friday afternoon.

John, Jeremy & Caius will all be attending the conference so make sure you say hi – and grab one of our always popular t-shirts too!

Experienced programmer Paul was a beta tester for Brightbox back in our very early days and has now come on board as our new senior developer. He’ll be joining Baz and Caius working on our customer-facing control panel and our behind-the-scenes systems, and he’ll also take a lead in project managing all the development side of things.

Welcome to the team, Paul!

The Control Panel that you may be familiar with uses Delayed Job. This is a Rails-specific gem that uses the database as a queue, with a nicely packaged worker process that handles messages as they arrive. Because the Control Panel only ever talks to Rails from Rails, this worked extremely well.

However, our other systems were not homogenous – there are a number of different interfaces that needed to be instructed at various times and across various machines, and Delayed Job didn’t really fit the bill. In particular, there were some tasks that could only happen on certain servers – while Delayed Job let us have multiple worker processes on different boxes, it essentially managed a single queue, so it could not differentiate between messages for one worker and messages for another.

AMQP to the rescue

Because of this, we looked elsewhere for our central work queue. We decided to use the AMQP protocol, with RabbitMQ as our implementation. AMQP is a protocol for taking incoming messages, placing them on one of many queues and routing them to a destination reliably; RabbitMQ is a server, built in Erlang, that is pretty lightweight and has some nice clustering features built-in.

Into the Warren

When it came to getting our Rails applications to talk to RabbitMQ, we had a couple of choices.

There is an AMQP gem which looked great. Unfortunately, as it uses EventMachine internally, it caused a few problems with our Passenger-based deployments – EventMachine opens a background thread which can cause issues to the web-server controlled Passenger processes.

There was also a gem called bunny. This works synchronously, thus avoiding the EventMachine issues – you connect and bunny either says “here’s the next message” or “the queue is empty” and then you disconnect.

It looked like bunny was perfect for our Passenger-based apps, with the AMQP gem being used in the non-Passenger side of things (actually in the end, we went with bunny on both sides, but for other reasons). But there was still a couple of things that we wanted to do with the messages.

Firstly, we are Rubyists and AMQP (the protocol) is language agnostic. In particular, we wanted to put a hash of data onto the queue and have a hash of data presented to us at the other end. Secondly, security is important to us – especially when it comes to tasks like rebooting or reconfiguring a server – so we wanted to ensure that the contents of the message were not tampered with in transit.

For these reasons, we built Warren. This adds filters over the AQMP layer. In our case, we had one filter that automatically marshalls the data to and from YAML as it is put onto and taken off the queue and a second HMAC filter that, with one line of configuration, ensures that the data was not altered on its journey.

Warren itself has an adapter layer, so it can quite happily use either the AMQP gem or bunny to actually talk to the queue. Adding new adapters, or filters, is as simple as creating a new subclass – Warren detects it and adds it to its stack.

Bigwig

With Warren it is pretty simple to put a message on to the queue:

Warren::Queue.publish(@queue_name, @hash_of_data)

It is just as easy to pull the messages off the other end:

Warren::Queue.subscribe(@queue_name) do | hash_of_data |
  do_something_with hash_of_data
end

But how do we implement this mysterious do_something_with(hash_of_data) call?

For our purposes, the endpoint needs to do different things depending upon which machine we are on and which queue we are listening to. The key one is our “builder” system which handles reboots, migrations and deployments, but there are several others, which while not yet live, will be coming onstream soon.

So effectively, we needed a set of daemon processes that listen to given queues and handle the messages that they receive. This splits into two components – the daemon framework and the workers themselves – which lead to Bigwig (keeping the rabbit theme going, Bigwig was a character in Watership Down).

Bigwig expects a configuration file, telling it how to connect to the AMQP server and a folder full of plugins. Each plugin is a simple Ruby class that performs a given action. Bigwig takes the incoming messages, looks for a parameter within the Hash called “method” and then uses that to find a plugin. The plugin is then invoked and can perform whatever tasks it needs to, using any data from the incoming message.

For monitoring purposes, Bigwig automatically deals with any incoming messages with a method of “ping” – these simply write a message to the log file. We can then send ping messages to the queue at set intervals and monitor that the log file is updated when expected – if we don’t see the update then we trigger an alert, in case the queue, or Bigwig, is down.

If the queue itself goes down, for whatever reason, then Bigwig uses an intelligent reconnect pattern. At first, it retries often, to minimise the downtime, but if the queue stays down, it reconnects less frequently, giving the queue a chance to restart before being swamped with connections. Of course, if the queue fails, then the ping messages won’t make it through and our monitor should alert us.

Calling back

Our primary internal application creates a Task object, with a unique identifier when dispatching tasks across the queue. This ID is then passed onto the queue and each Bigwig plugin is told of it when it is invoked. The application itself makes the Tasks available over ActiveResource.

This means that the Bigwig plugins themselves don’t actually need data passing over the queue. Instead, when they are invoked, they call back into the primary application, using the Task ID as the key to their ActiveResource call. They can then extract the data they need from the Task object and use that to do whatever they need to do. When completed, they call the complete! method on the ActiveResource object and our application knows that the job has been done (alternatively, they can call error! and our app knows that something went wrong). All of these appear in an event feed on the application’s homepage – so we know at a glance what activity is happening across our systems and, more importantly, what has failed.

We could have implemented this using AMQP again – a ‘responses’ queue that our primary application listens to, with each worker putting a message back when it was done. But we chose to use ActiveResource, partly because it is so simple (Rails makes it super-easy to expose your objects as XML and ActiveResource makes it equally easy for other Ruby programs to call back to them), and partly for philosophical reasons. When the primary application places a message onto the queue, it doesn’t necessarily know how it will be handled. AMQP dispatches it and a listener deals with it – in our case the listener is likely to be a Bigwig instance, but doesn’t have to be. But when the callback is made, it is always made into our application, which is already geared up for completing tasks or marking them as having an error. As we know that it is a Rails app at the other end, we chose to follow the path of least-resistance and use the “Rails way”.

However, we did have one issue with ActiveResource. Most of our plugins actually shell out to invoke Ruby scripts, with the plugin unpacking the information from the Task and generating command line arguments. This is a slight security risk, just because it is feasible that someone could inject a malicious command into the command line parameters (although as this is all internal it should never be an issue and Ruby’s system command does escape any supplied parameters). So we are switching the scripts to accept environment variables, thus bypassing the shell and eliminating a whole category of potential attacks. Secondly, using ActiveResource in a non-Rails application caused some weird errors – but only when you tried to POST back to the ActiveResource object. It turns out that ActiveSupport was causing some shifting to occur in the background and that was knocking out some of our own code. The solution was simple – we had to ensure that the require 'activeresource' statement was the first thing in each script.

So that’s a quick peek into the workings of some of our internal systems. We put a lot of thought into how these different components should work together, as their reliability is fundamental to our success as a business. And while they are not infallible (nothing ever is) I think you will agree that they show that using Ruby and AMQP as the “glue” to tie different systems together is not only feasible, it’s actually quite easy.

You’ll work closely with our Technical Director and Senior Sys Admin to run our virtual hosting platform, our Ruby on Rails hosting clusters, our network and our MySQL databases. A full job description is below.

The role is full-time and you can work from home, from our office in Leeds, or a bit of both (the rest of us do a bit of both).

Send a hello, a CV and salary expectations to jobs at the Brightbox UK domain. CVs should be in an open format, preferably PDF or plain text. Closing date is 24th October 2009.

As always, recruitment agents should e-mail our special recruitment company email address: root@localhost

• More jobs available at Brightbox
• More about working for Brightbox

Linux Systems Administrator

Job Description

• Work under the Technical Director and with our other system admins to support and run:
• our customers’ Ruby on Rails hosting clusters
• our virtual hosting platform
• our MySQL systems
• our networking infrastructure
• our backup systems
• our network management and monitoring systems
• Provide excellent customer service and support
• Assist with technical sales
• Participate in an out-of-hours on-call rota
• Work from home or our office in Leeds, or a mixture of the two
• Occasional visits to our hosting data centres for new builds and hardware maintenance
• Update technical and process documentation

Qualifications

• 2+ years experience with GNU/Linux based systems
• Experience with web technologies such as NGINX, Apache & mongrel
• Experience with Ruby, Rails desirable
• Familiarity with Puppet, Virtualization, MySQL, PostgreSQL, LVM desirable
• Familiarity with common Internet services such as DNS and SMTP
• Experience working with customers
• Ability to work both independently and as part of a team
• Keeps up with new technologies and developments
• Good written and verbal communication skills
• Able to commute to Leeds, UK for team meetings

All versions of Rails above and including version 2.0 are affected. There are two new official releases to fix this, 2.3.4 and 2.2.3.  If you’re still running Rails 2.0 or 2.1 and can’t upgrade, patches have been provided by the security team but need applying manually.  In this case, we’d recommend vendoring the rails gems and then applying the patches.

More details from the security team here.

You’ll work closely both with our Development and Technical Teams to develop our customer control panel and other backend systems. We’re looking for someone who, in addition to being a great coder, will be an integral part of the team and an organised and enthusiastic project manager.

The role is full-time and you can work from home, from our office in Leeds, or a bit of both (the rest of us do a bit of both).

Send a hello, a CV and salary expectations to jobs at the Brightbox UK domain. CVs should be in an open format, preferably PDF or plain text. Closing date is Friday 11th September 2009.

As always, recruitment agents should e-mail our special recruitment company email address: root@localhost

Job Description

• Work under the Technical Director and with our other developers to:
• Maintain and extend our “cloud” management systems
• Maintain and extend our billing system
• Build the front end for our next generation of deployment systems
• Help with application consultancy (advising customers how to design and scale their apps)
• Work on side-projects that help the community and promote Brightbox (such as isitruby19.com)
• Extract code to be shared with the community as open source/free software
• Invent and develop new ideas for services and products
• Project manage several concurrent projects
• Work from home or our office in Leeds

Qualifications

• Great Ruby on Rails coding skills
• A track record of implementing live RoR sites
• Experience of test driven development using RSpec
• Experience of deploying Rails using tools such as Capistrano
• Experience of project management desirable
• Proactive developer able to work independently as well as part of the team
• Suited to working in an agile, flexible environment
• Good written and verbal communication skills
• Living within 2 hours of the UTC time zone

The well-being of the Rails community is important to us, so we’d like to try and do what we can to keep it bubbling along nicely :)

One of the ideas that came up was some sort of Rails training network that would help bridge any skills gap. But before we do anything, we want to understand whether there is actually a skills gap, or just an “expectation gap” in either price or experience (anybody with any sort of Rails commercial experience is almost certainly gainfully employed at a good rate at the moment).

So, we’d like to hear from you if…

1) You’re in the market for Rails talent and would either be interested in using a Rails training programme, or interested in talking to ‘graduates’ of such a programme or

2) You’re a developer and you’d be interested in signing up for a Rails training programme and potentially getting your name in front of Rails employers.

Drop us an email to hello [at] the brightbox uk domain – we’ll gauge interest and take it from there.

It’s largely just a maintenance release, which means no Capistrano v2.0 support just yet, it still requires Capistrano 1.4.1. Cap 2.0 support is in the works though and will be released soon.

A quick summary of the changes:

• Fixes default Monit config bug (changes totalmem to mem, no more false memory overage alarms!)
• Fails out if MySQL database exists on cold_deploy (you really shouldn’t be running a cold_deploy a second time anyway)
• Adds reconfigure task (for easy adding of mongrels or domain changes)
• Archives Apache configs on cold_deploy or reconfigure
• Fixes Capistrano 1.4 dependency (will pull in v1.4.1 if necessary now)
• Changes default Apache log location to /var/log/web (for new cold_deploys or reconfigures)

Remember that you need to upgrade the Brightbox gem both in your development environment and on your Brightbox.

The HOWTO on deploying your Rails app using the Brightbox is here as usual.

Ubuntu’s release schedule sees a new version released roughly every 6 months. These releases contain the very latest versions of the software packaged with it and are supported for only 18 months. Once in a while a version is selected as Long Term Support release (LTS) which gets 5 years of server support (3 years for desktop). By support, I mean the Ubuntu team are committed to releasing security upgrades in a timely manner. Dapper was the first LTS version and is support through to June 2011.

If we’d chosen Edgy, security upgrades wouldn’t be available to us after April 2008, forcing all of our Brightboxers to upgrade to Feisty, and so on every 18 months. With the speed that the Rails community jump deployment strategy ships some might say this isn’t a problem, but most installations do need long term stability and Dapper provides that.

There are some issues though, mainly that Dapper’s version of Apache is too old to support the nice proxy balancing stuff that’s used for Mongrel deployments¹. To solve this, we chose to use a backported Apache package². This does mean that we have to commit to backporting all security fixes, but this is trivial compared to all our guest machines needing a full upgrade every 18 months. We still get the Ubuntu team working for us on the other 99.9% of packages.

For our users who like to ride the bleeding edge, they can still upgrade to Feisty themselves if they know what they are doing but for most, this isn’t what Brightbox is all about.

¹ A beta tester pointed us in the direction of this bug report requesting an official Apache backport for Dapper. The more people testing these packages and voicing their support, the more likely this might happen.

² We’re using the backported Apache package provided by kodefoo.com at the moment (http://www.kodefoo.com/2007/2/18/deploying-rails-on-ubuntu-dapper/) but are ready to roll our own if necessary.

1. Be developing in Ruby on Rails almost exclusively for new projects.
2. Currently hosting, or looking to host in the next 3 months, at least one production rails app
3. At least some experience of deploying rails to a production environment
4. Be the decision-maker, have the casting vote or at least have a major say in where apps are hosted at their company or agency
5. Have an active or rails-focused blog

Interested? drop a quick email to beta-req@brightbox.co.uk and tell us a bit about your app/apps and how you fit into the above profile. You might just get a free brightbox to play with, plus a nice discount when we launch.

UPDATE: (30 Jul 2007)
We’re well into the beta programme now so we’re not looking for any more beta testers.