Are you hosting your production MongoDB instances on Amazon AWS? At Scalegrid.io we manage hundreds of production MongoDB instances on AWS and have learnt a few things along the way. Here are a set of 10 questions you need to ask yourself and answer as you continue to manage your deployment. Almost all of the information below is applicable to other cloud service providers as well.
1. What is your high availability (HA) plan?
If you are using a single instance it might be time to look at replica sets. When using replica sets take care to ensure that you deploy each replica set in a different availability zone.
2. What is your disaster recovery (DR) plan?
If you are deploying all your entire replica sets in one region what happens when an entire AWS region melts down as it happened in April 2011? You might want to look into distributing your replica sets across regions.
3. Have you tested your DR plan?
Simulate machine, network and disk failures to understand your cluster behavior under failure conditions. You don’t want to encounter your first failover in production.
4. Are you backing up your instances?
Yes you need backups even if you have replica sets. Backups are necessary to deal with accidental erasure or when a new version of your app corrupts all your data. Make sure you are backing up regularly – preferably every few hours. You can backup from the secondary so that there is not a big impact on the primary. For a general overview of MongoDB backup options refer to our blog post - Mongodb backup options
5.Do your backup’s work?
Have you tried a recovery? How long does it take to recover and have all your replica’s resync? If you don’t know the answer now is a good time to do a dry run and try an end to end recovery.
6. How do you test application upgrades with production data?
One of the trickiest parts of application upgrade is testing with existing production data. Build a sequestered production like environment in which you can test your application upgrades with production data.
7. What sort of EBS volumes are you using?
If you are using standard EBS volumes consider switching to the new provisioned IOPS volumes. It is a little bit more expensive but worth every penny. You will see a lot less fluctuation in IO performance and sleep easier through the night.
8. Have you benchmarked the performance of your MongoDB instances?
If you haven’t you can benchmark using mongoperf or the Yahoo cloud server benchmark (YCSB). It’s good to know what your getting from your databases.
9. How do you monitor your instances?
If you are not monitoring your instances now would be a good time to start. 10gen has a freely available Mongo Monitoring Service (MMS) that you can start using to monitor your mongo clusters.
10. Are you exposing to your databases to the internet?
Today’s powerful CPUs and password cracking tools will crack open your password in a matter of a few hours. Use Amazon security groups to lock down access to your database and only give your front/mid tier access to the DB.
At Scalegrid.io we have helped answer a number of these questions for our customers. We provide a single click deployment of Mongo replica sets across availability zones or regions. We have automated backup and recovery process. We only use provisioned IOPS and provide easy ways for our customers to benchmark Mongo and also simulate failover in mongo clusters. If you have other questions/comments or feature requests we would love to hear from you. You can email us at firstname.lastname@example.org