Join the Brightbox team! 12 May 09

Brightbox is looking for two new people to join the systems team, a Senior Linux Systems Administrator and a Linux/Rails System Support person. Both roles are full-time and you can work from home, from our office in Leeds, or a bit of both.

Send a hello, a CV and salary expectations to jobs at the Brightbox UK domain. CVs should be in an open format, preferably PDF or plain text. Closing date is 31^st May 2009.

As always, recruitment agents should e-mail our special recruitment company email address: root@localhost

Read the rest of this entry »

Brightbox SAN storage quotas doubled! 16 Dec 08

Last week we announced a tasty increase in bandwidth allocations for all Brightbox virtual machine products. Today, we’re just as pleased to announce that we’re also doubling the SAN storage allocation of all Brightbox products…

• Brightbox 256 – was 5GB now 10GB!
• Brightbox 512 – was 10GB now 20GB!
• Brightbox 1GB – was 15GB now 30GB!
• Brightbox 2GB – was 20GB now 40GB!

Check out the updated product matrix for full specs.

Don’t forget, our storage isn’t just any old storage! Brightboxes are spread across multiple SANs, each of which have 14 disks in RAID10 (with hot spares of course :)

Also, our storage network is independent of the Xen host servers so should a problem occur with a Xen host, we can migrate customers to another node almost instantly.

The new storage allocations are available immediately on all new Brightboxes. Existing customers simply need to submit a request via the Helpdesk and we’ll schedule the storage increase (a short amount of downtime will be required to pick up the new storage allocation).

Emergency Maintenance, 8th – 11th July 2008 7 Jul 08

We’ll be doing some emergency maintenance on the cluster this week. We’ll be be migrating Brightboxes around to avoid having to reboot any of them, but the migration currently causes 10 to 20 seconds network outage.  Everything should continue as normal once networking is restablished – apps will not have to be restarted.

This work will take place between 1am and 3am every morning this week. We will need to log in to all boxes to make some Xen specific config changes during the migration. We’ll be contacting customers with HA clusters to arrange node failovers.

Apologies for the short notice.

37signals (and DHH) love Xen 4 Jan 08

37signals, the team behind popular productivity apps such as Basecamp, Backpack and Campfire, recently posted some stats which make fascinating reading. Of greater interest to us, though, is their use of Xen virtualisation for parts of the server infrastructure and the plans to upgrade more servers to Xen over the next couple of months, reducing their physical server count from 30 to 16 whilst remaining at around 100 cores.

DHH, Mr Rails himself, in another comment says of Xen:

Wonderful system. Strongly recommended for anyone building out a cluster.

What better recommendation to you need that Xen is a perfect solution for Rails hosting and scaling?!

Secure virtual disk deletion – is your data safe? 4 Dec 07

Everyone knows the dangers of old hard disks being discarded with sensitive data still on them, but what about virtual disks? With so many virtual machine hosting services cropping up of late (hi!), have you ever wondered what happens to your data when you delete your virtual machine?

Usually your machine’s ‘partition’ is just a small part of a larger disk array; the partition is deleted and the space returned for the pool to be used by another virtual machine. This means, the next time someone buys a virtual machine with the same host, some of the blocks that made up your filesystem could end up making up their filesystem. The metadata will be wiped clean when the filesystem is formatted of course, so they won’t just see your files listed, but the blocks can still contain your data. It depends on how they’re managing their disks.

Homework: go buy a virtual machine somewhere and pipe the contents of your new disk through the strings command and look out for anything that isn’t yours (ssh root@newmachine "dd if=/dev/sda1 bs=1M | strings"). Extra credit if you don’t get thrown off your new host on the first day for maxing out the disk IO :)

So, you’re probably careful and securely wipe your sensitive data before you leave, phew. But disk space is virtualised too. The blocks that make up your disk might not all be in order or even all be on the same disk. With snapshots, your data may exist in duplicate too that you can’t even access. And what about if you bought extra disk space, then removed it?

At Brightbox we use Linux’s LVM implementation to manage disk space and these are problems we have to deal with and we take it seriously. All virtual machine disks are wiped at the block level when the machine is deleted or when a new machine is created. The only corner case we’re likely to run into is if a disk image is extended into space that had previously been used as a snapshot or as a disk that was shrunk. Luckily we don’t currently offer snapshots or disk shrinking but it’ll be something we’ll probably offer at some point, so we’ll have to address it then.

Why we chose Ubuntu Dapper Drake 3 Aug 07

We’ve had a few beta testers ask about why we chose Ubuntu 6.06 (Dapper Drake) as our primary Xen guest installation, as opposed to one of the more recent releases such as Edgy or Feisty. We chose it primarily because of its support contract.

Ubuntu’s release schedule sees a new version released roughly every 6 months. These releases contain the very latest versions of the software packaged with it and are supported for only 18 months. Once in a while a version is selected as Long Term Support release (LTS) which gets 5 years of server support (3 years for desktop). By support, I mean the Ubuntu team are committed to releasing security upgrades in a timely manner. Dapper was the first LTS version and is support through to June 2011.

If we’d chosen Edgy, security upgrades wouldn’t be available to us after April 2008, forcing all of our Brightboxers to upgrade to Feisty, and so on every 18 months. With the speed that the Rails community jump deployment strategy ships some might say this isn’t a problem, but most installations do need long term stability and Dapper provides that.

There are some issues though, mainly that Dapper’s version of Apache is too old to support the nice proxy balancing stuff that’s used for Mongrel deployments¹. To solve this, we chose to use a backported Apache package². This does mean that we have to commit to backporting all security fixes, but this is trivial compared to all our guest machines needing a full upgrade every 18 months. We still get the Ubuntu team working for us on the other 99.9% of packages.

For our users who like to ride the bleeding edge, they can still upgrade to Feisty themselves if they know what they are doing but for most, this isn’t what Brightbox is all about.

¹ A beta tester pointed us in the direction of this bug report requesting an official Apache backport for Dapper. The more people testing these packages and voicing their support, the more likely this might happen.

² We’re using the backported Apache package provided by kodefoo.com at the moment (http://www.kodefoo.com/2007/2/18/deploying-rails-on-ubuntu-dapper/) but are ready to roll our own if necessary.