MySQL slow query improvements 28 Jan 10

We’ve just pushed some updates to the MySQL slow query warning interface in the Brightbox control panel, which includes two main improvements.

Firstly, where the same (or very similar) query occurs more than once, they are aggregated together rather than being recorded as individual slow queries. The user interface displays the count so you can see exactly how many times this slow query occurred in the last 7 days.

Secondly, the full SQL query is now displayed, so customers now have more information with which to optimise their applications.

Shared MySQL improvements 23 Dec 09

Over the last few weeks we’ve been working on scaling our shared MySQL facilities. Until recently, we’ve been able to run a single (albeit hefty) shared MySQL cluster but due to growing demand we’ve needed to scale this up considerably. The main cluster has had some performance problems recently and, while some tuning and vertical scaling bought us some time (we more than doubled the resources of the main cluster), the real focus has been on horizontal scaling.

We’ve built a bunch of new master-master replicated pairs and our backend systems now distribute customers between them on sign-up. We’ve also been contacting some customers and moving them to new clusters, to relieve the pressure on the main cluster (customers with heavy requirements are still recommended managed dedicated clusters – these will be available to purchase simply as additional products soon). We’re using puppet to automate a lot of the setup of the new clusters and can deploy a new one, with monitoring and backups, very quickly.

This work has almost quadrupled the shared MySQL resources within the space of a couple of weeks, and provides a simple platform to continue scaling indefinitely. The decentralisation also makes some aspects of administration easier, such as arranging downtime for maintenance.

The visible differences are small: rather than everyone connecting to one address, sqlreadwrite.brightbox.net, each account needs to use the address provided in the control panel. The old sqlreadwrite.brightbox.net has become db01.mysql.vm.brightbox.net (the old name will of course continue to work indefinitely), and the new clusters are at db02.mysql.vm.brightbox.net, db03.mysql.vm.brightbox.net etc. Our wiki documentation has been updated to reflect this – customers on the old cluster don’t have to make any changes, it only really affects new customers and customers we’ve contacted to arrange a move.

We’ve also been working on improving the slow query logger to provide more useful results. Instead of reporting every slow query ever logged, it produces an intelligent summary of the week’s queries. This means when you see a slow query in the control panel, it means it’s shown up repeatedly and very likely needs attention, as opposed to queries that just happened to take longer than usual due to load on the cluster. We’ll be rolling this work out just after Christmas.

This work represents a big investment in our shared MySQL platform, which we know is invaluable to a lot of our customers, and allows us to keep growing without sacrificing performance.

Join the Brightbox team! 12 May 09

Brightbox is looking for two new people to join the systems team, a Senior Linux Systems Administrator and a Linux/Rails System Support person. Both roles are full-time and you can work from home, from our office in Leeds, or a bit of both.

Send a hello, a CV and salary expectations to jobs at the Brightbox UK domain. CVs should be in an open format, preferably PDF or plain text. Closing date is 31^st May 2009.

As always, recruitment agents should e-mail our special recruitment company email address: root@localhost

Read the rest of this entry »

MySQL Maintenance: Thur 17 April 22:00-23:00 8 Apr 08

We’ll be doing some maintenance work on the MySQL cluster next Thursday, the 17th April starting at 22:00 and taking around an hour. There will be a couple of seconds interruption to service at 22:00 as we switch between cluster nodes, then a few more seconds as we switch back at around 23:00.  We’ll keep you up to date as usual using the status feed.

The maintenance is to apply some software updates but we’re taking the opportunity to do some MySQL performance tuning at the same time.

Planned MySQL Maintenance: 6 Jan 2200hrs-2230hrs 31 Dec 07

We’ll be doing some work on the MySQL cluster on Sunday 6th January 2008 between 2200hrs and 2230hrs. We’re changing the way the clustering is managed to improve stability and also upgrading to fix some security bugs.During the window you may see two or three brief outages (less than 2 minutes at a time). We’ll keep the status page up to date as usual. If you experience any issues after the maintenance window has closed, please submit a support ticket and we’ll deal with it straight away.

Update: (7 Jan 2007) maintenance went well with only few seconds downtime.