Weekly product update: Upgrading to MongoDB 2.4.1

This week we focused on backend improvements to make Mailgun more performant, reliable and scalable. Lots of boring stuff, but hopefully you’ll feel the difference.

One of the things that we did in particular was upgrade from MongoDB version 2.2 to 2.4.1. We’re big fans of MongoDB here at Mailgun. The bigger “Why MongoDB” post is long overdue but in the meantime, here’s a little color around the experience upgrading to the latest MongoDB version. We also upgraded to the latest version of pymongo: the MongoDB Python driver.

Upgrading to MongoDB 2.4.1 from 2.2

We highly recommend this version bump to anyone who’s using MongoDB in production. While you can see the complete list of what’s new in 2.4.1 on 10Gen web site, below is the list of features and improvements that we found particularly beneficial during evaluation on our staging environment:

• New MongoClient and MongoReplicaSet classes. They introduce the new safe default: waiting for write acknowledgements.

• Bug fixes that have happened after the introduction of 2.4.

Our infrastructure engineers performed the upgrade on a live high-stress system without any issues.

Upgrading PyMongo version

Well, by default the tests won’t pass.  You’ll have to fix some import errors. For example, objectidmodule has moved from pymongo to bson package. Otherwise the upgrade is pretty safe because the new default behavior is only present in the new connection classes.

There is one serious caveat to be aware of: in this version 10Gen introduced the idea of Requests. The connection objects you get from pymongo are now bound to their calling thread by default. Our testing showed a performance degradation by a factor on 3! Your mileage may very, but either way, it’s a very serious change of default behavior, we wish 10Gen was more explicit about this.

Upgrading the servers

You should not be using MongoDB without replica sets. This gives you the ability to not only be resilient to hardware failures, but it also makes it possible to perform database version upgrades on the fly. These are the steps we follow:

• Obviously, make sure you have fresh backup of your data. We make backups by shutting down one of the slaves, compacting it’s database and sending the tar ball of the DB directory to Rackspace Cloud Files.

• Upgrade one slave at a time, while monitoring the status of a replica set.

• Finally, step down the master.

• Upgrade the master.

All in all, very boring and uneventful procedure. Just like we like it when it comes to databases.

Till next week.

Happy Emailing, The Mailgunners