Send a hello, a CV and salary expectations to jobs at the Brightbox UK domain. CVs should be in an open format, preferably PDF or plain text. Closing date is 31^st May 2009.

As always, recruitment agents should e-mail our special recruitment company email address: root@localhost

Senior Linux Systems Administrator

Job Description

• Work with the Technical Director to design, build and run:

  • our virtual hosting platform

  • our MySQL systems

  • our customers’ Rails clusters

  • our backup systems

  • our networking infrastructure

  • new “cloud computing” services

• Second line support, with some shared first line responsibilities

• Out of hours on-call shifts

• Working with customers

• Assisting with technical sales

• Work from home or our office in Leeds

Qualifications

• 5+ years professional experience managing GNU/Linux based systems

• 2+ years experience with web technologies

• Familiarity with virtualized storage and systems

• Experience with Ruby, Rails, Debian/Ubuntu, MySQL, PostgreSQL, Puppet desirable

• Ability to work both independently and as part of a team

• Keeps up with new technologies and developments

• Good written and verbal communication skills

• Living within 2 hours of the UTC time zone

Linux/Rails System Support

Job Description

• Work under the Technical Director and Senior Systems Administrator to support and run:

  • our customers’ Ruby on Rails hosting clusters

  • our virtual hosting platform

  • our MySQL systems

• First line support

• Out of hours on-call shifts

• Working with customers

• Work from home or our office in Leeds

Qualifications

• 2+ years experience with GNU/Linux based systems

• Experience with Ruby, Rails required

• Familiarity with Puppet, Virtualization, MySQL, PostgreSQL, LVM desirable

• Experience working with customers

• Good written and verbal communication skills

• Living within 2 hours of the UTC time zone

• Brightbox 256 – was 5GB now 10GB!
• Brightbox 512 – was 10GB now 20GB!
• Brightbox 1GB – was 15GB now 30GB!
• Brightbox 2GB – was 20GB now 40GB!

Check out the updated product matrix for full specs.

Don’t forget, our storage isn’t just any old storage! Brightboxes are spread across multiple SANs, each of which have 14 disks in RAID10 (with hot spares of course :)

Also, our storage network is independent of the Xen host servers so should a problem occur with a Xen host, we can migrate customers to another node almost instantly.

The new storage allocations are available immediately on all new Brightboxes. Existing customers simply need to submit a request via the Helpdesk and we’ll schedule the storage increase (a short amount of downtime will be required to pick up the new storage allocation).

This work will take place between 1am and 3am every morning this week. We will need to log in to all boxes to make some Xen specific config changes during the migration. We’ll be contacting customers with HA clusters to arrange node failovers.

Apologies for the short notice.

DHH, Mr Rails himself, in another comment says of Xen:

Wonderful system. Strongly recommended for anyone building out a cluster.

What better recommendation to you need that Xen is a perfect solution for Rails hosting and scaling?!

Usually your machine’s ‘partition’ is just a small part of a larger disk array; the partition is deleted and the space returned for the pool to be used by another virtual machine. This means, the next time someone buys a virtual machine with the same host, some of the blocks that made up your filesystem could end up making up their filesystem. The metadata will be wiped clean when the filesystem is formatted of course, so they won’t just see your files listed, but the blocks can still contain your data. It depends on how they’re managing their disks.

Homework: go buy a virtual machine somewhere and pipe the contents of your new disk through the strings command and look out for anything that isn’t yours (ssh root@newmachine "dd if=/dev/sda1 bs=1M | strings"). Extra credit if you don’t get thrown off your new host on the first day for maxing out the disk IO :)

So, you’re probably careful and securely wipe your sensitive data before you leave, phew. But disk space is virtualised too. The blocks that make up your disk might not all be in order or even all be on the same disk. With snapshots, your data may exist in duplicate too that you can’t even access. And what about if you bought extra disk space, then removed it?

At Brightbox we use Linux’s LVM implementation to manage disk space and these are problems we have to deal with and we take it seriously. All virtual machine disks are wiped at the block level when the machine is deleted or when a new machine is created. The only corner case we’re likely to run into is if a disk image is extended into space that had previously been used as a snapshot or as a disk that was shrunk. Luckily we don’t currently offer snapshots or disk shrinking but it’ll be something we’ll probably offer at some point, so we’ll have to address it then.

Ubuntu’s release schedule sees a new version released roughly every 6 months. These releases contain the very latest versions of the software packaged with it and are supported for only 18 months. Once in a while a version is selected as Long Term Support release (LTS) which gets 5 years of server support (3 years for desktop). By support, I mean the Ubuntu team are committed to releasing security upgrades in a timely manner. Dapper was the first LTS version and is support through to June 2011.

If we’d chosen Edgy, security upgrades wouldn’t be available to us after April 2008, forcing all of our Brightboxers to upgrade to Feisty, and so on every 18 months. With the speed that the Rails community jump deployment strategy ships some might say this isn’t a problem, but most installations do need long term stability and Dapper provides that.

There are some issues though, mainly that Dapper’s version of Apache is too old to support the nice proxy balancing stuff that’s used for Mongrel deployments¹. To solve this, we chose to use a backported Apache package². This does mean that we have to commit to backporting all security fixes, but this is trivial compared to all our guest machines needing a full upgrade every 18 months. We still get the Ubuntu team working for us on the other 99.9% of packages.

For our users who like to ride the bleeding edge, they can still upgrade to Feisty themselves if they know what they are doing but for most, this isn’t what Brightbox is all about.

¹ A beta tester pointed us in the direction of this bug report requesting an official Apache backport for Dapper. The more people testing these packages and voicing their support, the more likely this might happen.

² We’re using the backported Apache package provided by kodefoo.com at the moment (http://www.kodefoo.com/2007/2/18/deploying-rails-on-ubuntu-dapper/) but are ready to roll our own if necessary.