Now, whenever a new box is deployed, the SSH keys of all the technical contacts on your Brightbox account are automatically pre-installed for the default ‘rails’ user. To add your own SSH key, edit your user profile within the control panel and paste in your SSH public key using the editor at the bottom.

Remember, this only affects newly provisioned boxes – as the it’s done during the box build stage. Removing or adding technical contacts in the control panel at a later date will not automatically change any access control on existing boxes.

We’ve been developing a more robust solution for users and accounts for some time and hope to launch this new system in the next few weeks. In this post, I’d like to give a sneak preview on the forthcoming changes…

• Each person will have a Brightbox user account
• We’ll be introducing a number of “roles” (tbc)
• Each account will always have an “owner” contact, this person is ultimately responsible for the account
• Existing Brightbox account holders will be migrated to a new owner contact
• The account owner will be able to invite additional users to their account
• The control panel will enable users with access to multiple accounts to easily move between the accounts to which they have access

Over the next couple of weeks, all customers will receive an email explaining the changes and the migration process.

nginx … is vulnerable to a buffer underflow when processing certain HTTP requests. An attacker can use this to execute arbitrary code with the rights of the worker process  or possibly perform denial of service attacks by repeatedly crashing worker processes via a specially crafted URL in an HTTP request.

New versions of our nginx packages that address this security vulnerability are now available.  nginx 0.6.39 (with the fair balancer module) is available from the Brightbox apt repositories – running the following command will get you the latest version:

sudo apt-get update
sudo apt-get install nginx

Our more experimental nginx-brightbox package has also been upgraded to 0.6.39.  This includes a number of nginx addons, such as the upload module, geoip module, and Phusion Passenger 2.0.5.

All versions of Rails above and including version 2.0 are affected. There are two new official releases to fix this, 2.3.4 and 2.2.3.  If you’re still running Rails 2.0 or 2.1 and can’t upgrade, patches have been provided by the security team but need applying manually.  In this case, we’d recommend vendoring the rails gems and then applying the patches.

More details from the security team here.

A denial of service (DoS) vulnerability was found on the BigDecimal standard library of Ruby. Conversion from BigDecimal objects into Float numbers had a problem which enables attackers to effectively cause segmentation faults.

ActiveRecord relies on this method, so most Rails applications are affected by this. Though this is not a Rails-specific issue.

We’re currently  building new Ruby packages for Brightbox customers with the relevant patches to fix this vulnerability. We’ll keep this post updated with the latest news.

UPDATE, 15:46 BST: New Ruby EE packages are now available in our Ruby Enterprise Ubuntu repository. We’re working on updates for the standard Ubuntu version of Ruby.

You can confirm that the update fixes the bug with the following command:

ruby -e 'require "bigdecimal";BigDecimal("E99999999").to_s("F");puts "OK"'

If your version of Ruby is vulnerable, you’ll get a “Segmentation fault” error message, otherwise it prints “OK”.

UPDATE: Official Ubuntu packages to fix this vulnerability are now available.  The Hardy package is libruby1.8 version 1.8.6.111-2ubuntu1.3 and the Dapper package is libruby1.8 version 1.8.4-1ubuntu1.7.  The packages will be available for install after a normal apt-get update.

For those that don’t know, Rails generates an authentication token within your forms and verifies this token when the form is submitted back to your application. This prevents attackers from crafting malicious requests whilst pretending to be your authenticated user.

However, for certain types of request (supposedly those that cannot be generated from a browser) this authentication token is ignored – in order to make it simpler for automated API access to your application (using JSON, XML or a few other data transport types). Unfortunately, text/plain is wrongly included as one of these types.

Luckily, the fix is simple. The long-term solution is to upgrade your application to Rails 2.1.3 or 2.2.2 (when they are released); the quick fix is even easier – tell Rails to verify text/plain requests by creating a file (called mime_type_csrf_fix.rb) in your config/initializers folder:


# temporary fix for http://www.rorsecurity.info/journal/2008/11/19/circumvent-rails-csrf-protection.html
Mime::Type.unverifiable_types.delete(:text)


The Ruby details are here: http://www.ruby-lang.org/en/news/2008/08/23/dos-vulnerability-in-rexml/ and the instructions for applying the monkey patch fix for Rails is on the security google group: http://groups.google.com/group/rubyonrails-security/browse_thread/thread/9fb60a1e22a88d30/330bcb96de877996#330bcb96de877996

Just to reiterate – this fault will affect the majority of Rails applications across all versions of Rails, whether you think you are processing XML or not, and we would urge all our customers (and all Rails users for that matter) to patch their applications until there is an upgrade to the Ruby libraries correcting the error.

Details of the bugs have been kept officially secret but people are figuring it out for themselves (thanks to Zed in particular).  This secrecy has just contributed to the fear, uncertainty and doubt surrounding the issues and hasn’t helped the situation at all.

We currently recommend sitting tight until proper fixes are available.  When this happens, distros will release new packages in the usual manner.  Brightboxes are based on the Ubuntu distro and their security team are aware of the problem and are working on it (see the bug status here).

For those of you using the standard Ruby from Dapper (most of you) you should be able to just upgrade  using aptitude as soon as Ubuntu release new packages.  For those of you using the backported Ruby 1.8.6 packages, you’ll need to wait for us to backport the fixes once they’re released.  We’ll obviously be doing this asap.

We’ll update the blog as we know more.

UPDATE: Ubuntu have fixed ruby1.8 packages available now. They have already appeared in the Ubuntu security repository and are available for install.  Preliminary testing of the Dapper packages has been successful (gems with native libraries too).  We’re re-backporting the Hardy 1.8.6 packages right now and they’ll be available soon.

UPDATE: We have the fixed Hardy packages (1.8.6-p111) backported to Dapper available on the Brightbox testing apt repository.  They’ve passed a lot of preliminary tests but have not been tested extensively in production yet.  Please report any problems  with them (segfaults etc.) to support@brightbox.co.uk.

One of our admin keys, used for accessing customer machines for support, was generated on a vulnerable version of Ubuntu.  This key is installed on Brightboxes by default though is limited to access from the private network only, mitigating the risk somewhat.

We’ve generated a new key and have now installed it on all affected Brightboxes and removed the weak one (you may have noticed some ssh connections from the private network to your box this morning as the user bbox-admin).

The Brightbox distribution is based on Ubuntu Dapper, which is not directly vulnerable to this bug, but if you are using an ssh key generated on one of the vulnerable versions then your Brightbox might be at risk.  If you use any of the vulnerable versions of Ubuntu yourself then please follow the instructions in the Ubuntu security notice.

Update: (7 Jan 2007) maintenance went well with only few seconds downtime.

Usually your machine’s ‘partition’ is just a small part of a larger disk array; the partition is deleted and the space returned for the pool to be used by another virtual machine. This means, the next time someone buys a virtual machine with the same host, some of the blocks that made up your filesystem could end up making up their filesystem. The metadata will be wiped clean when the filesystem is formatted of course, so they won’t just see your files listed, but the blocks can still contain your data. It depends on how they’re managing their disks.

Homework: go buy a virtual machine somewhere and pipe the contents of your new disk through the strings command and look out for anything that isn’t yours (ssh root@newmachine "dd if=/dev/sda1 bs=1M | strings"). Extra credit if you don’t get thrown off your new host on the first day for maxing out the disk IO :)

So, you’re probably careful and securely wipe your sensitive data before you leave, phew. But disk space is virtualised too. The blocks that make up your disk might not all be in order or even all be on the same disk. With snapshots, your data may exist in duplicate too that you can’t even access. And what about if you bought extra disk space, then removed it?

At Brightbox we use Linux’s LVM implementation to manage disk space and these are problems we have to deal with and we take it seriously. All virtual machine disks are wiped at the block level when the machine is deleted or when a new machine is created. The only corner case we’re likely to run into is if a disk image is extended into space that had previously been used as a snapshot or as a disk that was shrunk. Luckily we don’t currently offer snapshots or disk shrinking but it’ll be something we’ll probably offer at some point, so we’ll have to address it then.