Updated NGINX Passenger packages will follow shortly (they’ll be available via a separate ppa)

Updated NGINX Passenger packages will follow shortly (they’ll be available via a separate ppa)

We’ve also fixed the dependency problems that occurred in the past, where a newer version of Passenger broke the older NGINX packages. These NGINX packages now strictly depend on the Passenger packages, and we’ve put them in their own Launchpad archive so they’re always guaranteed to work (even if they ever lag behind the Apache packages).

You can add the PPA and install NGINX like this:

sudo apt-add-repository ppa:brightbox/passenger-nginx
sudo apt-get update
sudo apt-get install nginx-full

You’ll then need to enable the Passenger module, which can be usually done like this:

cat <<EOF > /etc/nginx/conf.d/passenger.conf
passenger_root /usr/lib/phusion-passenger;
EOF

We’ve also added our other Apache Passenger packages to Launchpad too (with Hardy support), just use:

sudo apt-add-repository ppa:brightbox/passenger

We’ll keep our own apt repository in sync with Launchpad for Hardy and Lucid too, in case you prefer that (use the passenger-nginx component for the nginx passenger packages).

Happy Passengering! (I’m pretty certain that’s not a real verb. It might be a proper verb. I’m pretty certain there is no such thing as a proper verb).

I don’t have to tell you why IPv6 is important – even the mainstream media has finally realised that the pool of IPv4 addresses is exhausted (it’s not like we needed tarot cards to predict it!). Any modern network must have support for IPv6.

We’ve been concentrating on perfecting IPv4 in Brightbox Cloud as that’s the immediate need, but we’ve considered IPv6 since day one of our network designs (over a year ago now!).  Every cloud server will get IPv6 addresses by default (and AAAA dns records of course), so you’ll be able to access your servers directly. We already have this working for our own test servers and we’ll have it in the hands of our customers very soon.

IPv6 will be a “first class citizen” in the Brightbox Cloud – supported throughout all of our services: Our Cloud Servers, Load Balancers, Firewall, Cloud IPs and of course our API. We’re currently working on the full implementation but I thought it was worthwhile announcing our plans, since this is such an important aspect of our service.

In the three and a half years since Brightbox started, we’ve built a solid reputation for our “Serious Rails Hosting” platform. We’ve spent the last twelve months building Brightbox Cloud our new cloud Infrastructure-as-a-Service which is already making big waves. We’re now looking for an experienced software engineer to join our small but highly productive development team. This is an awesome opportunity to use your skills and experience to help shape our future products and direction in this exciting and fast growing industry.

Main responsibilities:

• Lead development of our distributed cloud management and automation systems
• Break down complex projects into clear tasks and specs
• Proactively find ways of improving our services, systems and code
• Efficiently manage your own workload as well as that of others
• Extract code to be released as free software projects

The ideal candidate:

You have an insatiable desire to understand things, to break them apart and to fix them. You are continuously looking for ways to improve things. You are enthusiastic about some technologies and vocalise your reasoned dislike for others. You have strong opinions and stand by them when it really matters, but are willing to compromise when it doesn’t. You are able to communicate your ideas succinctly with eloquence and clarity. You have probably contributed to some free software projects, and perhaps started your own. You don’t take yourself too seriously and are comfortable with others reviewing your work. You love what you do.

Skills & Experience:

• Several years experience of coding in Ruby
• Experience with Ruby web frameworks, such as Rails, Sinatra
• Experience of testing frameworks, such as rspec
• Experience of deployment using tools, such as Capistrano
• In-depth knowledge of distributed cloud infrastructure concepts
• Experience programming asynchronous systems
• Experience consuming and creating APIs
• Experience of working in a fast-paced, agile environment
• Ability to empathise with users

Our team:

We’re Ruby developers and system engineers. We’re obsessive about great user experience and clear documentation.  We’re passionate about high availability, performance and consistency, though not all at once. We’ve coded seriously in Ruby, C, Delphi, Ada, C++, Python, Amiga AMOS, PHP, Java, Objective-C, Pascal, Eiffel, Basic, Perl, Informix 4GL, Javascript and Bash. We use Ubuntu, Centos and OS X. We take our work seriously but we have a lot of fun.

Like our systems, our teams are geographically distributed, communicating in real-time throughout the working day using our private Jabber conference or phone calls. We manage our work with Redmine, usually organising chunks of work into weekly sprints. We’re a lean organisation – you’ll have direct access to leadership as well as other teams. Our development, engineering and operations teams all work closely together which shortens the feedback loop and means we get things done better.

Location:

Remote, but ideally working within 2 hours of UTC.

Salary:

circa £55k p.a. (depending on experience).

Closing date:

Friday 22nd April 2011.

How to apply:

Send a hello email and a CV (PDF or plain text) to jobs@brightbox.co.uk. All correspondence from recruitment agents must be in Esperanto and to our special recruitment email address: root@localhost

$ brightbox-images list img-1okdf img-aoubd

id         owner      type      created_on  status  size   name
-----------------------------------------------------------------------------------------
img-1okdf  brightbox  official  2010-11-19  public  20480  FreeBSD 8.1 minimal (i686)
img-aoubd  brightbox  official  2010-11-19  public  20480  FreeBSD 8.1 minimal (x86_64)
-----------------------------------------------------------------------------------------

As you’d expect, you can control FreeBSD servers via the API, just like any other Brightbox cloud server – and that includes snapshotting the disk. If you’re a FreeBSD nerd and want to have a play, then sign up to our beta.

Now I’m off to learn how to use FreeBSD to see what all the fuss is about :)

We’ve now got Ubuntu Hardy and Lucid packages available on our apt repository and we consider it ready for production use. Brightbox customers can upgrade from Passengr 2.x simply by running these commands on their Brightboxes:

sudo apt-get update
sudo apt-get install -qy libapache2-mod-passenger

If you’ve not got a Brightbox, you’ll need to add our apt repository key and config first. You can read more about it on our wiki.

Brightbox’s birthday is a matter of days away and we’re starting our fourth year by increasing RAM for all Brightboxes by 50%. We’ll be upgrading existing Brightboxes over the next couple of weeks, though the product names in the control panel will change straight away.

MySQL connection quotas doubled

Since you’ll now be able to fit more app processes on your Brightboxes, you’ll need more MySQL connections, so we’re doubling those for all products too.

New Relic RPM for all

And as if this wasn’t enough, we’re expanding our New Relic RPM offering to include all Brightbox sizes. See our wiki page for more details about getting it set up.

We’re taking the opportunity to rename our products too, as naming by RAM size starts to get a bit ugly when you move beyond powers of two. The table below shows how the products are being renamed and what your new RAM size will be. Happy birthday us!

We’ve got some other big things in the pipeline too, so stay tuned. The next few months are going to be great.

Update (19 October 2010): We have now completed these upgrades for all customers. The extra MySQL connections are already available to everyone. Customers who have not rebooted their Brightboxes recently will need to do so in order to start using their extra RAM.

We now have Passenger 3.0.0-pre3 packages available for Ubuntu 10.04 (Lucid) and Ubuntu 8.04 (Hardy). As they’re pre-release versions, we don’t recommend them in production just yet and have put them in their own repository to prevent any accidental upgrades.

If you’re not already a Brightbox customer, then you’ll need to set up base access to our apt repository first.

Otherwise, just add the new passenger-testing repository (switch “lucid” to “hardy” if you’re on Hardy):

sudo sh -c 'echo "deb http://apt.brightbox.net lucid passenger-testing" > /etc/apt/sources.list.d/brightbox-passenger-testing.list'

Then you can upgrade/install 3.0.0-1bbox1~pre3

apt-get update
apt-get install libapache2-mod-passenger

Passenger now has a native library, which depends on your version of ruby. For simplicity, these packages currently require the ruby1.8 packages to be installed. You can, of course, switch to ruby1.9 and passenger will auto-compile the necessary native support for you. We’ll be providing packages for 1.9 support soon, so you won’t need to rely on the auto-compiling.

Now, whenever a new box is deployed, the SSH keys of all the technical contacts on your Brightbox account are automatically pre-installed for the default ‘rails’ user. To add your own SSH key, edit your user profile within the control panel and paste in your SSH public key using the editor at the bottom.

Remember, this only affects newly provisioned boxes – as the it’s done during the box build stage. Removing or adding technical contacts in the control panel at a later date will not automatically change any access control on existing boxes.

We’ve built our Ubuntu Hardy packages for i386 and AMD64 architectures which are now available from the Brightbox apt repository.  We’ve also upgraded our librack-ruby packages to 1.1.0.

For example:

  require 'xapian-fu'
  include XapianFu
  db = XapianDb.new(:dir => 'example.db', :create => true,
                    :store => [:title, :year])
  db << { :title => 'Brokeback Mountain', :year => 2005 }
  db << { :title => 'Cold Mountain', :year => 2004 }
  db << { :title => 'Yes Man', :year => 2008 }
  db.flush
  db.search("mountain").each do |match|
    puts match.values[:title]
  end

The full announcement is here, github project here and rdoc here.  Hope you find it useful!

This package won’t actually support Rails 3 just yet though as we’ve yet to package and test the librack 1.1.0, which Rails 3 depends on.  We’re working on it now and once we’re happy with it we’ll add new packages to our repository as usual.

Brightbox is also sponsoring the meeting so there will be free pizza afterwards (free as in pizza, not as in speech).

More details on the NWRUG blog page.

Hope to see you there!

We’ve built a bunch of new master-master replicated pairs and our backend systems now distribute customers between them on sign-up. We’ve also been contacting some customers and moving them to new clusters, to relieve the pressure on the main cluster (customers with heavy requirements are still recommended managed dedicated clusters – these will be available to purchase simply as additional products soon). We’re using puppet to automate a lot of the setup of the new clusters and can deploy a new one, with monitoring and backups, very quickly.

This work has almost quadrupled the shared MySQL resources within the space of a couple of weeks, and provides a simple platform to continue scaling indefinitely. The decentralisation also makes some aspects of administration easier, such as arranging downtime for maintenance.

The visible differences are small: rather than everyone connecting to one address, sqlreadwrite.brightbox.net, each account needs to use the address provided in the control panel. The old sqlreadwrite.brightbox.net has become db01.mysql.vm.brightbox.net (the old name will of course continue to work indefinitely), and the new clusters are at db02.mysql.vm.brightbox.net, db03.mysql.vm.brightbox.net etc. Our wiki documentation has been updated to reflect this – customers on the old cluster don’t have to make any changes, it only really affects new customers and customers we’ve contacted to arrange a move.

We’ve also been working on improving the slow query logger to provide more useful results. Instead of reporting every slow query ever logged, it produces an intelligent summary of the week’s queries. This means when you see a slow query in the control panel, it means it’s shown up repeatedly and very likely needs attention, as opposed to queries that just happened to take longer than usual due to load on the cluster. We’ll be rolling this work out just after Christmas.

This work represents a big investment in our shared MySQL platform, which we know is invaluable to a lot of our customers, and allows us to keep growing without sacrificing performance.

sudo apt-get update
sudo apt-get install libapache2-mod-xsendfile
sudo a2enmod xsendfile
sudo invoke-rc.d apache reload

As usual, details on installing the packages from our repository are available on our wiki.

If you’re using Passenger and it’s making you happy, please do consider supporting its development by donating money in the form of an “Enterprise License” direct from Phusion, the company behind it.

Expiring the cache is just a case of deleting the html file. But what if you want to expire an entire tree of cache files? Say you change something in a header or footer, so every single page needs expiring at once.

The usual way to do this is to just delete the entire page cache tree, with FileUtils.rm_rf.  This works pretty well, but with a big tree you’ll get strange behaviour under high load due to concurrent access.  Whilst your rm_rf process is deleting the tree, file by file, your webserver will still be looking in there for page cache files and Rails will still be trying to write them.

This is easily solvable.  On a POSIX compliant filesystem, like EXT3, the rename operation is atomic – it either happens or it doesn’t, there is no in-between state where it is half renamed or anything.  So, before running the rm_rf, you rename your highest-level cache directory to something temporary.  This means the cache expiry is instantaneous, even if you have 100 meg of page cache, and you won’t get Rails writing new page cache files into it whilst you delete it.

It’s a good idea to use a robust temporary filename format so two processes don’t end up renaming a cache directory to the same thing at the same time, especially if your page cache is on a shared filesystem.

An example snippet of code follows. It assumes you want to expire pages from the controller named entries.

require 'socket'
tmp_cache_dir = [Socket.gethostname, Process.pid, Time.now.to_i, rand(0xffff)].to_s
page_cache_tree = File.join(ApplicationController.page_cache_directory, 'entries')
FileUtils.mv(page_cache_tree, page_cache_tree + tmp_cache_dir)
FileUtils.rm_rf(page_cache_tree + '-' + tmp_cache_dir)

This renames RAILS_ROOT/public/entries to something like RAILS_ROOT/public/entries-hostname22351125837834752773 (which should be sufficiently unique across a number of nodes in a cluster as to avoid collisions) and then deletes it.

If you want to expire the entire page cache, you’ll need to change the default from RAILS_ROOT/public as you can’t rename and delete that (it has images and javascripts etc. too!). Change it to something like RAILS_ROOT/public/page_cache. You’ll need to update your web server config to consider this new path too.

Remember that rename is only atomic within the same filesystem, so if you symlink your page cache directory from your RAILS_ROOT onto a shared filesystem, then you need to do all your renames and deletes within this.

Also, this has the side effect of working around a bug with our shared filesystem, GlusterFS, which got upset with multiple concurrent directory tree deletes (this is now fixed though).

As with our other Ruby EE packages, they upgrade (i.e replace) the standard 1.8 Ruby installation. This means all your gems stay the same, and everything on your system immediately starts using them (Phusion’s own Ubuntu packages do not work like this).  We’ve tested it with the usual Railsy native gems, RMagick, Mongrel, fasthread etc. and have had no problems.

If you’re on a Brightbox, just edit /etc/apt/sources.list.d/brightbox-rubyee.list and change the rubyee component to rubyee-testing:

deb http://apt.brightbox.net/ hardy rubyee-testing

Then update and upgrade:

sudo apt-get update
sudo apt-get install libruby1.8

If you’re not on a Brightbox, see the instructions on our wiki first. The wiki also documents how to revert back to the old packages.

As said above, we now have 64bit packages available too (which was recently made easier by some Debian package dependency updates, also included in our repository).

Please let us know how they worked out for you on our forum. Thanks!

nginx … is vulnerable to a buffer underflow when processing certain HTTP requests. An attacker can use this to execute arbitrary code with the rights of the worker process  or possibly perform denial of service attacks by repeatedly crashing worker processes via a specially crafted URL in an HTTP request.

New versions of our nginx packages that address this security vulnerability are now available.  nginx 0.6.39 (with the fair balancer module) is available from the Brightbox apt repositories – running the following command will get you the latest version:

sudo apt-get update
sudo apt-get install nginx

Our more experimental nginx-brightbox package has also been upgraded to 0.6.39.  This includes a number of nginx addons, such as the upload module, geoip module, and Phusion Passenger 2.0.5.

All versions of Rails above and including version 2.0 are affected. There are two new official releases to fix this, 2.3.4 and 2.2.3.  If you’re still running Rails 2.0 or 2.1 and can’t upgrade, patches have been provided by the security team but need applying manually.  In this case, we’d recommend vendoring the rails gems and then applying the patches.

More details from the security team here.

Passenger 2.2.4 actually is just a small bug fix release for a memory leak in 2.2.3, but obviously brings all the benefits of 2.2.3 too.  A huge number of bugs have been fixed, particularly the “Broken Pipe” errors some sites under heavy loads were experiencing.

As usual, details on installing the packages from our repository are available on our wiki.

If you’re using Passenger and it’s making you happy, please do consider supporting its development by donating money in the form of an “Enterprise License” direct from Phusion, the company behind it.

A denial of service (DoS) vulnerability was found on the BigDecimal standard library of Ruby. Conversion from BigDecimal objects into Float numbers had a problem which enables attackers to effectively cause segmentation faults.

ActiveRecord relies on this method, so most Rails applications are affected by this. Though this is not a Rails-specific issue.

We’re currently  building new Ruby packages for Brightbox customers with the relevant patches to fix this vulnerability. We’ll keep this post updated with the latest news.

UPDATE, 15:46 BST: New Ruby EE packages are now available in our Ruby Enterprise Ubuntu repository. We’re working on updates for the standard Ubuntu version of Ruby.

You can confirm that the update fixes the bug with the following command:

ruby -e 'require "bigdecimal";BigDecimal("E99999999").to_s("F");puts "OK"'

If your version of Ruby is vulnerable, you’ll get a “Segmentation fault” error message, otherwise it prints “OK”.

UPDATE: Official Ubuntu packages to fix this vulnerability are now available.  The Hardy package is libruby1.8 version 1.8.6.111-2ubuntu1.3 and the Dapper package is libruby1.8 version 1.8.4-1ubuntu1.7.  The packages will be available for install after a normal apt-get update.