Vital Logs

We heavily use caching in our models using cache_fu.  However, we started to notice that a small number of objects in our system were out of sync between MySQL and memcached.
In one case, this led to model validation exceptions as a user appeared unregistered in memcache, but when we went to register the the user the db would complain that the user is already registered.

What was the bug?

Lets look at the log file…

Ex) user.save!
BEGIN
Update (0.7ms) UPDATE `users` SET `token` = '665456562', `updated_at` = '2009-05-13 17:22:26' WHERE `id` = 4
MemCache Delete (0.000285)  users:2
COMMIT

Uh oh!
We use InnoDB in MySQL to get transaction support and the way rails implements ActiveRecord calls is that it wraps a transaction around everything within an ActiveRecord method call.  This includes the after_save callback to expire_cache.

This leads to the following scenario with two threads where one is reading and another writing:

t1: BEGIN t1
t1: update user in db
t1: expire user in memcache
t2: get user from memcache
t2: cache miss -> get user from db
t2: set user in memcache
t1: COMMIT

Since t2 is reading from the database before t1 has committed it will read the old value and save the old value into memcache. The db is updating, but memcache is out of sync.

What’s the solution?
We need a way for the expire_cache calls after save to be outside the transaction.  This way, t2’s read will either be before the db update and cache expiry or after the db update on the expired cache. The three possible cases are listed below. In each case, memcache and the database remain in sync.

Case 1: t2 is before the COMMIT
t1: BEGIN t1
t1: update user in db
t2: get user from memcache
t1: COMMIT
t1: expire user in memcache


Case 2: t2 is after the COMMIT, but before cache expiry
t1: BEGIN t1
t1: update user in db
t1: COMMIT
t2: get user from memcache
t1: expire user in memcache


Case 3: t2 is after the COMMIT and cache expiry
t1: BEGIN t1
t1: update user in db
t1: COMMIT
t1: expire user in memcache
t2: get user from memcache
t2: cache miss -> get user from db
t2: set user in memcache (new user)


Luckily, there is a great plug-in called after_commit that does what we need. This plugin implements additional callbacks like “after_commit” which run after save, but are outside the commit.