New: Automatically add SSH keys to new Brightboxes 3 Jun 10

Getting access to your newly purchased Brightboxes used to require a trip to the control panel to retrieve the ‘rails’ user password. If you’re using config management systems like Chef or Puppet this is likely the only laborious aspects of configuring your box. No longer!

Now, whenever a new box is deployed, the SSH keys of all the technical contacts on your Brightbox account are automatically pre-installed for the default ‘rails’ user. To add your own SSH key, edit your user profile within the control panel and paste in your SSH public key using the editor at the bottom.

Remember, this only affects newly provisioned boxes – as the it’s done during the box build stage. Removing or adding technical contacts in the control panel at a later date will not automatically change any access control on existing boxes.

Preview: Multiple users and accounts 15 Jan 10

Right now, each Brightbox customer has a single username/password to access their Brightbox account. For many smaller customers, this works fine and they can easily operate the various aspects of their account themselves e.g manage virtual machines, billing and support tickets. However, there are also a number of scenarios where this doesn’t work too well – especially when there are multiple people managing one or more accounts.

We’ve been developing a more robust solution for users and accounts for some time and hope to launch this new system in the next few weeks. In this post, I’d like to give a sneak preview on the forthcoming changes…

• Each person will have a Brightbox user account
• We’ll be introducing a number of “roles” (tbc)
• Each account will always have an “owner” contact, this person is ultimately responsible for the account
• Existing Brightbox account holders will be migrated to a new owner contact
• The account owner will be able to invite additional users to their account
• The control panel will enable users with access to multiple accounts to easily move between the accounts to which they have access

Over the next couple of weeks, all customers will receive an email explaining the changes and the migration process.

NGINX buffer underflow security vulnerability 15 Sep 09

From the Debian Security team (CVE-2009-2629):

nginx … is vulnerable to a buffer underflow when processing certain HTTP requests. An attacker can use this to execute arbitrary code with the rights of the worker process  or possibly perform denial of service attacks by repeatedly crashing worker processes via a specially crafted URL in an HTTP request.

New versions of our nginx packages that address this security vulnerability are now available.  nginx 0.6.39 (with the fair balancer module) is available from the Brightbox apt repositories – running the following command will get you the latest version:

sudo apt-get update
sudo apt-get install nginx

Our more experimental nginx-brightbox package has also been upgraded to 0.6.39.  This includes a number of nginx addons, such as the upload module, geoip module, and Phusion Passenger 2.0.5.

Rails form helper security vulnerability 4 Sep 09

A vulnerability has been found in the Rails form helpers that allows an attacker to inject arbitrary HTML into pages.  This opens up an unpatched Rails app to potential cross site scripting attacks (XSS), which could result in stolen session cookies and other such scenarios.

All versions of Rails above and including version 2.0 are affected. There are two new official releases to fix this, 2.3.4 and 2.2.3.  If you’re still running Rails 2.0 or 2.1 and can’t upgrade, patches have been provided by the security team but need applying manually.  In this case, we’d recommend vendoring the rails gems and then applying the patches.

More details from the security team here.

Ruby BigDecimal denial of service 10 Jun 09

From ruby-lang.org:

A denial of service (DoS) vulnerability was found on the BigDecimal standard library of Ruby. Conversion from BigDecimal objects into Float numbers had a problem which enables attackers to effectively cause segmentation faults.

ActiveRecord relies on this method, so most Rails applications are affected by this. Though this is not a Rails-specific issue.

We’re currently  building new Ruby packages for Brightbox customers with the relevant patches to fix this vulnerability. We’ll keep this post updated with the latest news.

UPDATE, 15:46 BST: New Ruby EE packages are now available in our Ruby Enterprise Ubuntu repository. We’re working on updates for the standard Ubuntu version of Ruby.

You can confirm that the update fixes the bug with the following command:

ruby -e 'require "bigdecimal";BigDecimal("E99999999").to_s("F");puts "OK"'

If your version of Ruby is vulnerable, you’ll get a “Segmentation fault” error message, otherwise it prints “OK”.

UPDATE: Official Ubuntu packages to fix this vulnerability are now available.  The Hardy package is libruby1.8 version 1.8.6.111-2ubuntu1.3 and the Dapper package is libruby1.8 version 1.8.4-1ubuntu1.7.  The packages will be available for install after a normal apt-get update.

Rails CSRF Security Vulnerability 19 Nov 08

Users of Rails 2.1 and 2.2 need to be aware of a vulnerability in Rails’ CSRF forgery protection.

For those that don’t know, Rails generates an authentication token within your forms and verifies this token when the form is submitted back to your application. This prevents attackers from crafting malicious requests whilst pretending to be your authenticated user.

However, for certain types of request (supposedly those that cannot be generated from a browser) this authentication token is ignored – in order to make it simpler for automated API access to your application (using JSON, XML or a few other data transport types). Unfortunately, text/plain is wrongly included as one of these types.

Luckily, the fix is simple. The long-term solution is to upgrade your application to Rails 2.1.3 or 2.2.2 (when they are released); the quick fix is even easier – tell Rails to verify text/plain requests by creating a file (called mime_type_csrf_fix.rb) in your config/initializers folder:


# temporary fix for http://www.rorsecurity.info/journal/2008/11/19/circumvent-rails-csrf-protection.html
Mime::Type.unverifiable_types.delete(:text)


Rails security vulnerability 26 Aug 08

If you’ve been following the Rails security list you’ll know that there has been a serious flaw uncovered in the REXML library that allows an easy Denial of Service attack on the vast majority of Rails applications.

The Ruby details are here: http://www.ruby-lang.org/en/news/2008/08/23/dos-vulnerability-in-rexml/ and the instructions for applying the monkey patch fix for Rails is on the security google group: http://groups.google.com/group/rubyonrails-security/browse_thread/thread/9fb60a1e22a88d30/330bcb96de877996#330bcb96de877996

Just to reiterate – this fault will affect the majority of Rails applications across all versions of Rails, whether you think you are processing XML or not, and we would urge all our customers (and all Rails users for that matter) to patch their applications until there is an upgrade to the Ruby libraries correcting the error.

Ruby Security Vulnerabilities 25 Jun 08

Some of you will have noticed the kerfuffle regarding the recent Ruby security vulnerabilities.  Fixed version of Ruby were released over the weekend but they are causing crashes in applications.  Until working fixes are available we’re all a bit stuck.

Details of the bugs have been kept officially secret but people are figuring it out for themselves (thanks to Zed in particular).  This secrecy has just contributed to the fear, uncertainty and doubt surrounding the issues and hasn’t helped the situation at all.

We currently recommend sitting tight until proper fixes are available.  When this happens, distros will release new packages in the usual manner.  Brightboxes are based on the Ubuntu distro and their security team are aware of the problem and are working on it (see the bug status here).

For those of you using the standard Ruby from Dapper (most of you) you should be able to just upgrade  using aptitude as soon as Ubuntu release new packages.  For those of you using the backported Ruby 1.8.6 packages, you’ll need to wait for us to backport the fixes once they’re released.  We’ll obviously be doing this asap.

We’ll update the blog as we know more.

UPDATE: Ubuntu have fixed ruby1.8 packages available now. They have already appeared in the Ubuntu security repository and are available for install.  Preliminary testing of the Dapper packages has been successful (gems with native libraries too).  We’re re-backporting the Hardy 1.8.6 packages right now and they’ll be available soon.

UPDATE: We have the fixed Hardy packages (1.8.6-p111) backported to Dapper available on the Brightbox testing apt repository.  They’ve passed a lot of preliminary tests but have not been tested extensively in production yet.  Please report any problems  with them (segfaults etc.) to support@brightbox.co.uk.

Ubuntu Openssh vulnerability 14 May 08

A vulnerability in some versions of Openssh on Debian and Ubuntu Linux was announced yesterday which can result in attackers gaining ssh access to machines with weak keys.  Any versions of Openssh that can produce these weak keys needs to be upgraded, and any weak keys in use need to be regenerated.

One of our admin keys, used for accessing customer machines for support, was generated on a vulnerable version of Ubuntu.  This key is installed on Brightboxes by default though is limited to access from the private network only, mitigating the risk somewhat.

We’ve generated a new key and have now installed it on all affected Brightboxes and removed the weak one (you may have noticed some ssh connections from the private network to your box this morning as the user bbox-admin).

The Brightbox distribution is based on Ubuntu Dapper, which is not directly vulnerable to this bug, but if you are using an ssh key generated on one of the vulnerable versions then your Brightbox might be at risk.  If you use any of the vulnerable versions of Ubuntu yourself then please follow the instructions in the Ubuntu security notice.

Planned MySQL Maintenance: 6 Jan 2200hrs-2230hrs 31 Dec 07

We’ll be doing some work on the MySQL cluster on Sunday 6th January 2008 between 2200hrs and 2230hrs. We’re changing the way the clustering is managed to improve stability and also upgrading to fix some security bugs.During the window you may see two or three brief outages (less than 2 minutes at a time). We’ll keep the status page up to date as usual. If you experience any issues after the maintenance window has closed, please submit a support ticket and we’ll deal with it straight away.

Update: (7 Jan 2007) maintenance went well with only few seconds downtime.