Firstly, where the same (or very similar) query occurs more than once, they are aggregated together rather than being recorded as individual slow queries. The user interface displays the count so you can see exactly how many times this slow query occurred in the last 7 days.

Secondly, the full SQL query is now displayed, so customers now have more information with which to optimise their applications.

We’ve built a bunch of new master-master replicated pairs and our backend systems now distribute customers between them on sign-up. We’ve also been contacting some customers and moving them to new clusters, to relieve the pressure on the main cluster (customers with heavy requirements are still recommended managed dedicated clusters – these will be available to purchase simply as additional products soon). We’re using puppet to automate a lot of the setup of the new clusters and can deploy a new one, with monitoring and backups, very quickly.

This work has almost quadrupled the shared MySQL resources within the space of a couple of weeks, and provides a simple platform to continue scaling indefinitely. The decentralisation also makes some aspects of administration easier, such as arranging downtime for maintenance.

The visible differences are small: rather than everyone connecting to one address, sqlreadwrite.brightbox.net, each account needs to use the address provided in the control panel. The old sqlreadwrite.brightbox.net has become db01.mysql.vm.brightbox.net (the old name will of course continue to work indefinitely), and the new clusters are at db02.mysql.vm.brightbox.net, db03.mysql.vm.brightbox.net etc. Our wiki documentation has been updated to reflect this – customers on the old cluster don’t have to make any changes, it only really affects new customers and customers we’ve contacted to arrange a move.

We’ve also been working on improving the slow query logger to provide more useful results. Instead of reporting every slow query ever logged, it produces an intelligent summary of the week’s queries. This means when you see a slow query in the control panel, it means it’s shown up repeatedly and very likely needs attention, as opposed to queries that just happened to take longer than usual due to load on the cluster. We’ll be rolling this work out just after Christmas.

This work represents a big investment in our shared MySQL platform, which we know is invaluable to a lot of our customers, and allows us to keep growing without sacrificing performance.

Send a hello, a CV and salary expectations to jobs at the Brightbox UK domain. CVs should be in an open format, preferably PDF or plain text. Closing date is 31^st May 2009.

As always, recruitment agents should e-mail our special recruitment company email address: root@localhost

Senior Linux Systems Administrator

Job Description

• Work with the Technical Director to design, build and run:

  • our virtual hosting platform

  • our MySQL systems

  • our customers’ Rails clusters

  • our backup systems

  • our networking infrastructure

  • new “cloud computing” services

• Second line support, with some shared first line responsibilities

• Out of hours on-call shifts

• Working with customers

• Assisting with technical sales

• Work from home or our office in Leeds

Qualifications

• 5+ years professional experience managing GNU/Linux based systems

• 2+ years experience with web technologies

• Familiarity with virtualized storage and systems

• Experience with Ruby, Rails, Debian/Ubuntu, MySQL, PostgreSQL, Puppet desirable

• Ability to work both independently and as part of a team

• Keeps up with new technologies and developments

• Good written and verbal communication skills

• Living within 2 hours of the UTC time zone

Linux/Rails System Support

Job Description

• Work under the Technical Director and Senior Systems Administrator to support and run:

  • our customers’ Ruby on Rails hosting clusters

  • our virtual hosting platform

  • our MySQL systems

• First line support

• Out of hours on-call shifts

• Working with customers

• Work from home or our office in Leeds

Qualifications

• 2+ years experience with GNU/Linux based systems

• Experience with Ruby, Rails required

• Familiarity with Puppet, Virtualization, MySQL, PostgreSQL, LVM desirable

• Experience working with customers

• Good written and verbal communication skills

• Living within 2 hours of the UTC time zone

The maintenance is to apply some software updates but we’re taking the opportunity to do some MySQL performance tuning at the same time.

Update: (7 Jan 2007) maintenance went well with only few seconds downtime.