Archive: posts from December 2009

Shared MySQL improvements 23 Dec 09

Over the last few weeks we’ve been working on scaling our shared MySQL facilities. Until recently, we’ve been able to run a single (albeit hefty) shared MySQL cluster but due to growing demand we’ve needed to scale this up considerably. The main cluster has had some performance problems recently and, while some tuning and vertical scaling bought us some time (we more than doubled the resources of the main cluster), the real focus has been on horizontal scaling.

We’ve built a bunch of new master-master replicated pairs and our backend systems now distribute customers between them on sign-up. We’ve also been contacting some customers and moving them to new clusters, to relieve the pressure on the main cluster (customers with heavy requirements are still recommended managed dedicated clusters – these will be available to purchase simply as additional products soon). We’re using puppet to automate a lot of the setup of the new clusters and can deploy a new one, with monitoring and backups, very quickly.

This work has almost quadrupled the shared MySQL resources within the space of a couple of weeks, and provides a simple platform to continue scaling indefinitely. The decentralisation also makes some aspects of administration easier, such as arranging downtime for maintenance.

The visible differences are small: rather than everyone connecting to one address, sqlreadwrite.brightbox.net, each account needs to use the address provided in the control panel. The old sqlreadwrite.brightbox.net has become db01.mysql.vm.brightbox.net (the old name will of course continue to work indefinitely), and the new clusters are at db02.mysql.vm.brightbox.net, db03.mysql.vm.brightbox.net etc. Our wiki documentation has been updated to reflect this – customers on the old cluster don’t have to make any changes, it only really affects new customers and customers we’ve contacted to arrange a move.

We’ve also been working on improving the slow query logger to provide more useful results. Instead of reporting every slow query ever logged, it produces an intelligent summary of the week’s queries. This means when you see a slow query in the control panel, it means it’s shown up repeatedly and very likely needs attention, as opposed to queries that just happened to take longer than usual due to load on the cluster. We’ll be rolling this work out just after Christmas.

This work represents a big investment in our shared MySQL platform, which we know is invaluable to a lot of our customers, and allows us to keep growing without sacrificing performance.

Apache x-sendfile module for Ubuntu Hardy 22 Dec 09

We’ve just added Caspar Clemens Mierau‘s package for the Apache x-sendfile module to our Ubuntu Hardy package repositories, so now it’s trivially easy for Brightbox customers to start using it. We’ve also built an AMD64 version too.  Just install the package, enable it and reload Apache:

sudo apt-get update
sudo apt-get install libapache2-mod-xsendfile
sudo a2enmod xsendfile
sudo invoke-rc.d apache reload

New: Performance graphs (beta) 8 Dec 09

We quietly rolled out a new feature into the Brightbox control panel a couple of weeks ago, adding performance graphs for each of your Brightbox virtual machines. To view graphs for a virtual machine, click the “full details” link from the overview screen. You can currently choose to view CPU, disk i/o, public and private network usage for the last hour, day, week and month.

The graphing service is currently in beta, so you might find some occasional gaps in data for the time being, but over time this should prove a very useful feature.

Paul joins Team Brightbox! 7 Dec 09

We welcomed our latest addition to the Brightbox team a fortnight ago – Paul Thornthwaite.

Experienced programmer Paul was a beta tester for Brightbox back in our very early days and has now come on board as our new senior developer. He’ll be joining Baz and Caius working on our customer-facing control panel and our behind-the-scenes systems, and he’ll also take a lead in project managing all the development side of things.

Welcome to the team, Paul!