If you are running a MongoDB replica set in a public cloud environment for any reasonable length of time the odds are that you have experienced a ‘rollback’. It does sound daunting but there are simple steps to recover your data in case your system experiences a rollback.
When does a rollback occur?
A rollback in a MongoDB replica set can occur in the following sequence of steps
1. A MongoDB primary accepts writes that are not yet replicated to other secondaries and then crashes
2. Another server becomes primary and accepts other writes
3. When the previous primary comes back up again and resyncs its state to the majority
Can I prevent rollbacks from happening in the first place?
Sure. But as always it comes with a price. You can set your write concern to ‘MAJORITY’. This will require that all your writes are committed to a majority of nodes in the replica set before MongoDB can successfully acknowledge it. However, this will have a dramatic impact on your write throughput. So in the real world, a good balance might be to only use write concern ‘MAJORITY’ for writes of important transactional data
How to recover data from a rollback?
Below are four easy steps to recover your data in case of a rollback
1. Find the rollback files
When a rollback occurs the bson files of the rollback data are placed in the ‘rollback’ directory of your MongoDB data path. The files look something like is
<dbname>.<collectioname>.2016-02-08T19-34-44.0.bson
2. Load the data from the rollback into a separate database or server
My preference is to copy the rollback files to a new server and use mongorestore to load them into the server. Here is the syntax you can use
mongorestore -u <> -p <> -h 127.0.0.1 -d <rollbackrestoretestdb> -c <rollbackrestoretestc> <path to the .bson file> --authenticationDatabase=<database of user>
3. Sift through the data & cleanse unneeded data
At this point, as a database administrator, you will have to use your discretion to decide which data from the rollback you want to keep and which data no longer makes sense. In most cases its unlikely you can just import all data. This is the probably the most difficult step in rollback recovery.
4. Import data into primary of cluster
Use the mongodump and mongorestore tools to download the cleansed data and re-import them into your original production cluster.
For more information about the rollback operations please consult the official MongoDB documentation
