Expiring an entire page cache tree atomically 16 Nov 09

As you’ll all know, Rails has page caching baked right in – the first time an action is hit, it writes a html file of the result to the filesystem. Subsequent hits are served direct from the html file at high speed by the web server without ever involving your Rails app.

Expiring the cache is just a case of deleting the html file. But what if you want to expire an entire tree of cache files? Say you change something in a header or footer, so every single page needs expiring at once.

The usual way to do this is to just delete the entire page cache tree, with FileUtils.rm_rf.  This works pretty well, but with a big tree you’ll get strange behaviour under high load due to concurrent access.  Whilst your rm_rf process is deleting the tree, file by file, your webserver will still be looking in there for page cache files and Rails will still be trying to write them.

This is easily solvable.  On a POSIX compliant filesystem, like EXT3, the rename operation is atomic – it either happens or it doesn’t, there is no in-between state where it is half renamed or anything.  So, before running the rm_rf, you rename your highest-level cache directory to something temporary.  This means the cache expiry is instantaneous, even if you have 100 meg of page cache, and you won’t get Rails writing new page cache files into it whilst you delete it.

It’s a good idea to use a robust temporary filename format so two processes don’t end up renaming a cache directory to the same thing at the same time, especially if your page cache is on a shared filesystem.

An example snippet of code follows. It assumes you want to expire pages from the controller named entries.

require 'socket'
tmp_cache_dir = [Socket.gethostname, Process.pid, Time.now.to_i, rand(0xffff)].to_s
page_cache_tree = File.join(ApplicationController.page_cache_directory, 'entries')
FileUtils.mv(page_cache_tree, page_cache_tree + tmp_cache_dir)
FileUtils.rm_rf(page_cache_tree + '-' + tmp_cache_dir)

This renames RAILS_ROOT/public/entries to something like RAILS_ROOT/public/entries-hostname22351125837834752773 (which should be sufficiently unique across a number of nodes in a cluster as to avoid collisions) and then deletes it.

If you want to expire the entire page cache, you’ll need to change the default from RAILS_ROOT/public as you can’t rename and delete that (it has images and javascripts etc. too!). Change it to something like RAILS_ROOT/public/page_cache. You’ll need to update your web server config to consider this new path too.

Remember that rename is only atomic within the same filesystem, so if you symlink your page cache directory from your RAILS_ROOT onto a shared filesystem, then you need to do all your renames and deletes within this.

Also, this has the side effect of working around a bug with our shared filesystem, GlusterFS, which got upset with multiple concurrent directory tree deletes (this is now fixed though).