For further information on using these packages see the release announcement for our Ruby EE 2010.01 packages.

Available from today, you can now choose either Ubuntu Hardy or Lucid as the base OS when building new Brightboxes.

Ubuntu Lucid is the latest Long Term Support (LTS) version of Ubuntu with security fixes provided until April 2015. It brings a whole bunch of upgrades such as Ruby 1.8.7, Monit 5, Apache 2.2.14 and new packages like CouchDB, Sphinx, Chef, RabbitMQ, MongoDB, ejabberd and many more.

We’ve treated it to the usual Brightbox Ruby deployment tune-up, including our Ruby Enterprise Edition 1.8.7-2010.01 packages. Updated Phusion Passenger packages are now available on our newly Lucid-enabled apt repository.

When buying a new Brightbox, you’ll see a combo box that you can use to select Lucid (Hardy is still currently the default). Upgrading from Hardy to Lucid isn’t really viable due to the way Hardy boxes handle kernels, so you’ll either need to request a re-image (which involves wiping your box, so make backups!) or buy a new box and move your apps to it.

Now, whenever a new box is deployed, the SSH keys of all the technical contacts on your Brightbox account are automatically pre-installed for the default ‘rails’ user. To add your own SSH key, edit your user profile within the control panel and paste in your SSH public key using the editor at the bottom.

Remember, this only affects newly provisioned boxes – as the it’s done during the box build stage. Removing or adding technical contacts in the control panel at a later date will not automatically change any access control on existing boxes.

As with our other Ruby EE packages, they upgrade (i.e replace) the standard 1.8 Ruby installation. This means all your gems stay the same, and everything on your system immediately starts using them (Phusion’s own Ubuntu packages do not work like this).

These packages are also the best way to get Ruby 1.8.7 on Hardy, which you’ll need if you’re playing with Rails 3.

If you’re on a Hardy based Brightbox, just create or edit /etc/apt/sources.list.d/brightbox-rubyee.list to contain the rubyee-testing component like so:

deb http://apt.brightbox.net/ hardy rubyee-testing

If you’re on one of our Lucid beta boxes provisioned before today, simply create /etc/apt/sources.list.d/brightbox-rubyee.list and add the rubyee component:

deb http://apt.brightbox.net/ lucid rubyee

Finally, update and upgrade libruby1.8:

sudo apt-get update
sudo apt-get install libruby1.8 irb1.8 libopenssl-ruby1.8 libreadline-ruby1.8 rdoc1.8 ruby1.8

If you’re not on a Brightbox, see the instructions on our wiki first. The wiki also documents how to revert back to the old packages.

We’ll still be offering Hardy-based Brightboxes, of course.

We’re now looking for volunteers beta test Lucid Brightboxes. If you’d like one to test, please email lucidbeta@brightbox.co.uk.

Beta test Brightboxes are free, but Lucid isn’t finished and there are still bugs, so you shouldn’t plan on running your live customer-facing sites on one yet. Once we launch Lucid-based boxes for all, we’ll be switching off the beta boxes.

Space on the test platform is limited, so don’t delay! There are Brightbox-goodies on offer for the best bug reports. We’ll be giving priority to existing Brightbox customers.

ActiveMessaging is a great piece of code, but at first glance it appears to have a couple of wrinkles

• The standard configuration doesn’t support load balancing (so that you can talk to another broker if your normal one isn’t responding).
• The queues and topics you want to speak to are defined at class level – which is a problem if you want the queues to be dynamically defined based upon data values.

But thanks to Ruby, YAML and a bit of lateral thinking you can get around these issues.

ActiveMessaging uses the Stomp Gem under the hood, and that can do load balancing by itself. So all you have to do is get ActiveMessaging to pass the relevant incantations down to the Stomp Gem – intact. And you do that with a bit of fancy YAML. The trick is to add your host configurations to the login configuration entry in hash format like this:

 production:
    adapter: stomp
    login:
      :randomize: true
      :hosts:
      - :login: myloginid1
        :passcode: mypassword1
        :host: broker-host-1.somedomain.co.uk
      - :login: myloginid2
        :passcode: mypassword2
        :host: broker-host-2.somedomain.co.uk

The Stomp Connection class checks login to see if it is a hash and if it is uses that in preference to anything else. You can setup anything Stomp understands within that Hash. Check the class code for details.

The beauty of this approach is that it is entirely in the configuration. Your Rails code doesn’t need to know about it. The reconnection happens automatically in the background.

I’ve used an observer class within Rails to handle the message sending. It watches a model and sends messages when stuff is created and updated – nicely separating the concerns. But I wanted to talk to different queues depending upon what was in the model. So the standard ActiveMessaging::MessageSender approach didn’t seem appropriate.

After a bit of digging around it turns out that you can get hold of the broker connection directly rather than going via the Gateway.

client = ActiveMessaging::Gateway.connection

Then you can send to any queue you like, not just the ones defined up front in messaging.rb:

client.send("/queue/#{some_variable}", body, headers)

This is a lower level connection and you lose the filter chains and some error checking. However if you need it, it’s there.

This is generally considered to be best practice where ever a server is geographically located, however, it may  cause a problem if you are hosting a  geographically specific web application.

If your web application requires a different time-zone such as CET, BST or PST, or Daylight Saving Time is giving you a headache,  take a look at our Wiki page for a timely introduction to the subject of time-zones on Brightboxes.

Again, we’re sponsoring the event and have provided hosting for the website. Our technical director John Leach is also speaking at the conference – his talk “UNIX: Rediscovering the wheel” will take place in the Great Hall on Friday afternoon.

John, Jeremy & Caius will all be attending the conference so make sure you say hi – and grab one of our always popular t-shirts too!

We’ve built our Ubuntu Hardy packages for i386 and AMD64 architectures which are now available from the Brightbox apt repository.  We’ve also upgraded our librack-ruby packages to 1.1.0.

Maintenance period

Starting at 12:00 (midday) on Weds 17 Feb, the Brightbox control panel will be unavailable for approximately 20 minutes whilst we perform the updates. One of the updates will include a migration to create an owner user account (name and email address) for all existing active customer accounts.

Change of login from “username” to email address

Once the above maintenance period is completed, you’ll be able to login to the control panel again as normal, but now using your registered email address to login, together with the existing password.

Adding and inviting users

Once logged in you’ll be able to give other people access to your account via the “User access” link.

It’s important to note that initially, as described below, all users that you give access to your account will have full access (apart from the ability to invite other users).

Interim billing and technical roles

Currently, customers are able to specify two additional “contacts” on their account – a billing contact (receives cc’ed payment email notifications) and a technical contact (receives copies of support ticket notifications and server notifications). These “old style” contacts do not have access to the account, unless the account owner supplies them with the account username and password.

The main purpose of this work is to enable a “many to many” relationship between users and accounts i.e multiple users can access a single account and single users can have multiple accounts. We will be introducing a more extensive permissions system into the control panel in due course, but in the meantime the four “roles” that users can have will each have full access to the account but differ in their behaviour…

• Owner (one per account): Full access + add/invite new users to have access. The owner role is the person ultimately responsible for the account and can’t be assigned by users.
• Primary technical (one per account): Full access + single point of contact for Brightbox technical support to receive email notifications etc
• Technical (many per account): Full access.
• Billing (many per account): Full access + receive copies of payment email notifications

If you have any questions about these changes and how they will affect you, please drop us an email.

For example:

  require 'xapian-fu'
  include XapianFu
  db = XapianDb.new(:dir => 'example.db', :create => true,
                    :store => [:title, :year])
  db << { :title => 'Brokeback Mountain', :year => 2005 }
  db << { :title => 'Cold Mountain', :year => 2004 }
  db << { :title => 'Yes Man', :year => 2008 }
  db.flush
  db.search("mountain").each do |match|
    puts match.values[:title]
  end

The full announcement is here, github project here and rdoc here.  Hope you find it useful!

Firstly, where the same (or very similar) query occurs more than once, they are aggregated together rather than being recorded as individual slow queries. The user interface displays the count so you can see exactly how many times this slow query occurred in the last 7 days.

Secondly, the full SQL query is now displayed, so customers now have more information with which to optimise their applications.

Today I’m pleased to announce a new revision of the Brightbox Deployment Gem, version 2.3.6. This release fixes some of the issues reported to us by our users, along with a few extra features.

The documentation on the wiki has been updated with the new fixes and features. If you have any tips or find any errors just let us know.

NEW FEATURES

Apt Package Dependencies

Just as you can currently define gems your application relies on and have them automatically installed onto your servers, you can now do the same with apt packages. Define them in your deploy.rb and they’ll be installed before installing your required rubygems, which means you can use it to install apt packages that gems depend on. As an example, the nokogiri gem depends on a couple of apt packages.

depend :remote, :apt, "libxml2-dev"
depend :remote, :apt, "libxslt1-dev"
depend :remote, :gem, "nokogiri", ">= 0"

Intermediate SSL Certificates

Support for Intermediate SSL Certificates in Apache has been added, meaning you no longer need to manually edit the Apache config files directly. Just add one line to your deploy.rb and the gem now takes care of updating the Apache config for you. An example SSL configuration:

set :ssl_certificate, "my_cert.pem"
set :ssl_key, "my_cert.key"
set :ssl_intermediate, "intermediate.crt"

See the wiki page for Adding SSL Support to Apache for more information.

Deploy.local.rb

A common issue we’ve seen with is setting the deploy password. If you put it in deploy.rb, then it inevitably ends up in your source control repository, which is a bad idea. A workaround we’ve suggested in the past is to have a deploy.local.rb file, which isn’t tracked by source control, and load that from within your deploy.rb file. This allows you to have the password (or any other sensitive settings) defined, but kept outside your source control.

As part of this release, the gem now has official support for a deploy.local.rb file. If it exists, the gem will load it in for you.

For more information see the Advanced Gem Settings page on the wiki.

Disable (re)generating webserver config

If you’ve made manual changes to your webserver (apache or nginx) configuration, then you don’t want an accidental deploy:setup to overwrite your configs. Up until now you just had to avoid running the command, but now there’s a setting to disable it for you.

set :generate_webserver_config, false

FIXES AND ENHANCEMENTS

• Max Age setting now accepts a number as well as a string
• deploy:initial reloads Apache so the new config is loaded

We’ve been reasonably resistant to publishing an SLA until now, not because of doubts over our infrastructure or ability to deliver, but because of the minimal value we felt it would add for customers in the event of actual downtime. We’re a pragmatic bunch at Brightbox and like to avoid adding things just for the sake of it or “because everyone else does” :)

However, and it’s a pretty big “however”, we understand that for many people an SLA or “uptime guarantee” is a useful gauge of whether a provider is actually willing to “put their money where their mouth is” and make a financial commitment to meeting a specific minimum target.

So, I’m pleased to announce that from 1 Jan 2010 (backdated) Brightbox offers a 99.95% SLA on virtual machine and load balancing products (see Terms and Conditions for details). We’ll likely expand the SLA soon to also cover our MySQL products and other products currently in the pipeline, but we’ll need a different method of assessing “availability” for these products.

We’ve been developing a more robust solution for users and accounts for some time and hope to launch this new system in the next few weeks. In this post, I’d like to give a sneak preview on the forthcoming changes…

• Each person will have a Brightbox user account
• We’ll be introducing a number of “roles” (tbc)
• Each account will always have an “owner” contact, this person is ultimately responsible for the account
• Existing Brightbox account holders will be migrated to a new owner contact
• The account owner will be able to invite additional users to their account
• The control panel will enable users with access to multiple accounts to easily move between the accounts to which they have access

Over the next couple of weeks, all customers will receive an email explaining the changes and the migration process.

This package won’t actually support Rails 3 just yet though as we’ve yet to package and test the librack 1.1.0, which Rails 3 depends on.  We’re working on it now and once we’re happy with it we’ll add new packages to our repository as usual.

Brightbox is also sponsoring the meeting so there will be free pizza afterwards (free as in pizza, not as in speech).

More details on the NWRUG blog page.

Hope to see you there!

We’ve built a bunch of new master-master replicated pairs and our backend systems now distribute customers between them on sign-up. We’ve also been contacting some customers and moving them to new clusters, to relieve the pressure on the main cluster (customers with heavy requirements are still recommended managed dedicated clusters – these will be available to purchase simply as additional products soon). We’re using puppet to automate a lot of the setup of the new clusters and can deploy a new one, with monitoring and backups, very quickly.

This work has almost quadrupled the shared MySQL resources within the space of a couple of weeks, and provides a simple platform to continue scaling indefinitely. The decentralisation also makes some aspects of administration easier, such as arranging downtime for maintenance.

The visible differences are small: rather than everyone connecting to one address, sqlreadwrite.brightbox.net, each account needs to use the address provided in the control panel. The old sqlreadwrite.brightbox.net has become db01.mysql.vm.brightbox.net (the old name will of course continue to work indefinitely), and the new clusters are at db02.mysql.vm.brightbox.net, db03.mysql.vm.brightbox.net etc. Our wiki documentation has been updated to reflect this – customers on the old cluster don’t have to make any changes, it only really affects new customers and customers we’ve contacted to arrange a move.

We’ve also been working on improving the slow query logger to provide more useful results. Instead of reporting every slow query ever logged, it produces an intelligent summary of the week’s queries. This means when you see a slow query in the control panel, it means it’s shown up repeatedly and very likely needs attention, as opposed to queries that just happened to take longer than usual due to load on the cluster. We’ll be rolling this work out just after Christmas.

This work represents a big investment in our shared MySQL platform, which we know is invaluable to a lot of our customers, and allows us to keep growing without sacrificing performance.

sudo apt-get update
sudo apt-get install libapache2-mod-xsendfile
sudo a2enmod xsendfile
sudo invoke-rc.d apache reload

The graphing service is currently in beta, so you might find some occasional gaps in data for the time being, but over time this should prove a very useful feature.

Experienced programmer Paul was a beta tester for Brightbox back in our very early days and has now come on board as our new senior developer. He’ll be joining Baz and Caius working on our customer-facing control panel and our behind-the-scenes systems, and he’ll also take a lead in project managing all the development side of things.

Welcome to the team, Paul!

Brightbox is sponsoring the event and John will be speaking on Thursday about the wonders of UNIX – how its often forgotten tools can help with Ruby on Rails development and deployment.

They’ll also have some of our highly coveted tshirts and stickers to give away so if you’re going, make sure you track them down and say hello.

As usual, details on installing the packages from our repository are available on our wiki.

If you’re using Passenger and it’s making you happy, please do consider supporting its development by donating money in the form of an “Enterprise License” direct from Phusion, the company behind it.

However, this session data is never deleted, which means your session table continues to grow and grow forevermore. Your old sessions are left stored in the database, and although the table is indexed to help with finding sessions, it will eventually fill the disk up.

At this point you might be thinking the solution is obvious, just empty the sessions table after so long and let it fill up again. Rails even provides a rake task that does this for us, rake db:sessions:clear. The problem with taking this approach is that any active sessions get lost as well, which could be people with items in their baskets, currently logged in users, etc.

There is another solution, which is to only delete sessions that we consider to no longer be active. The updated_at column in the sessions table has an index on it, and thus looks designed for this type of query to be run. On the Brightbox control panel, we’ve decided this is session data that hasn’t been updated for over 24 hours. (A side effect of this is customers that haven’t visited the control panel in the last 24 hours are logged out. We’ve decided this is ok, but YMMV of course.)

We use the following rake task that clears out sessions 24 hours or older, which is run via cron at 3am every morning. You can change the threshold by editing "1 DAY" in the query, see the MySQL DATE_ADD() docs for valid values.

desc "Clear expired sessions"
task :clear_expired_sessions => :environment do
    sql = 'DELETE FROM sessions WHERE updated_at < DATE_SUB(NOW(), INTERVAL 1 DAY);'
    ActiveRecord::Base.connection.execute(sql)
end