General backup strategy for MongoDB

MongoDB backup is actually a basic operation, has been asked recently, it seems that many people are not familiar with the place. To avoid repeating the explanation over and over again, a blog is summed up as a post for later review. Please correct me if there are some mistakes.

1. Built-in method 1.1 copy database files

It doesn't have to be explained much, it's almost useful for any database, simple and rude. But like most databases, this operation must be done in case the Mongod instance is stopped to ensure that you get the correct state of the database. Otherwise, if there is a write operation during the backup process, it may cause the backed up library to be in an unhealthy state and unusable. The need to stop the database has resulted in a very low availability of this method and limited usage scenarios.

1.2 Mongodump/mongorestore

I'm not going to dwell on how these two commands are used, because the articles that introduce the two commands are already on the web, and you can easily find them elsewhere or in official documents. Another reason not to dwell on it is also to hope to see the readers here can develop an independent problem-solving ability, do not rely on the so-called "master", the problem of independent thinking on the independent solution, which will be one of the necessary skills in your future.

Here are some things that others have said relatively little.

1.2.1 Besides Mongodump/mongorestore, there's a couple of combinations .Mongoexport/Mongoimport

Where is the difference?

1. Mongoexport/mongoimport Import/Export is in JSON format, and Mongodump/mongorestore Import/export is Bson format.
2. JSON is readable but bulky, and Bson is a binary file, small in size but hardly readable for humans.
3. Between some MongoDB versions, the Bson format may differ depending on the version, so mongodump/mongorestore between versions may not be successful, depending on compatibility between versions. When you cannot use Bson for cross-version data migrations, using the JSON format, Mongoexport/mongoimport, is an option. Cross-version Mongodump/mongorestore individuals are not recommended, so please check the documentation first to see if the two versions are compatible (most of the time Yes).
4. Although JSON has a good cross-version commonality, it retains only the data part, not the index, the account and other basic information. You should pay attention when using.

In short, these two sets of tools in the actual use of each have advantages, should be based on the application scenario selection (as if not said). But strictly speaking, the main role of Mongoexport/mongoimport is to import/export data when used, not a real backup tool. So there is no introduction here.

1.2.2 Mongodump There is a worth mentioning option is--oplog

Note that this is replica set or Master/slave mode dedicated (standalone mode running MongoDB is not recommended).

--oplog use Oplog for taking a point-in-time snapshot

Look at the English description good Cow b appearance, Point-in-time snapshot Oh, the first time I saw this sentence the understanding is that it can let the database back to any point in time in the state of the time, the United States for a good burst. But not really. The actual effect is to generate a Oplog.bson file at the same time as the export, storing all the oplog between the dump and dump end you start . This thing is specific what to sell first Xiaoguanzi. Use a graph to illustrate the coverage of the next Oplog.bson:

In order to explain in the back not to say dizzy, further explained before the first explanation of what is oplog and its related concepts. The official documentation has been fully explained, click to view Chinese documents or English documents.

Simply put, in replica set Oplog is a set of capacity (Capped collection) whose default size is 5% of disk space (which can be modified by the--OPLOGSIZEMB parameter). Db.oplog.rs, located in the local library, is interested to see what's inside. It records all changes (insert/update/DELETE) operations of the database over time for the entire Mongod instance. When space runs out of time, the oldest record is automatically overwritten. So from the timeline, the coverage of the Oplog is roughly the same:

Its coverage is called the oplog time window . It is important to note that because Oplog is a set of constant capacity, the range of time windows can cover varies depending on the number of updates in your unit time. To view the current Oplog time window estimates, you can use the following command:

Test:primary>rs.printreplicationinfo () configured Oplog size:1561. 5615234375MB <--Collection Size log length start to End:423849secs (117. 74hrs) <--estimated window coverage time Oplog first event Time: Wed Sep the  -  -: the: -gmt+0800(CST) Oplog LastEvent Time: Mon Sep -  -  the: at: -gmt+0800(CST) Now:mon Sep -  -  -:Panax Notoginseng: -gmt+0800(CST)

Oplog has a very important feature- idempotent (idempotent). That is, for a data set, the result will be the same, no matter how many times it is replayed, using the operations recorded in Oplog. For example, if an insert operation is recorded in Oplog, there are two identical records in the database because you have replayed it two times. This is a very important feature, and it is the basis for the subsequent operations.

go back to the subject and see what Oplog.bson really does. The first thing to understand is that the data is dependent on each other, such as set a in the order, set B to store all the details of the order, then only one order has a complete breakdown of the correct state. It is assumed that at any point in time, the data of A and B sets is fully corresponding and meaningful (it is not easy for a non-relational database to do this, and the data structure is not reasonable for mongodb.) But here we assume that this condition is true, then if a is at point X, and B is at a point-in-time y after X, you can imagine that the data in a and B are very likely to lose meaning.

Come back to see Mongodump's operation. In the process of Mongodump, the database is not locked to ensure that the entire library freezes at a fixed point in time, which is often not allowed in business. So the final result of dump is that a set is the state of 10 o'clock, and the B set is the state of 10 points and 1 minutes. Such a backup, even if restored back, can be imagined results are limited. Then the meaning of the above oplog.bson is reflected here. If you re-do all of the operations recorded in Oplog again on the basis of the dump data, the data can represent the database state at the point in time (Point-in-time) at the end of the dump.

The important condition of this conclusion is idempotent: The existing data, redo Oplog will not repeat, the non-existent data redo Oplog can enter the database. So when the oplog is done to a certain point in time, the database reverts to the state at that point in time.

Take a look at the options for Mongorestore. Options related to Oplog are --oplogreplay and --oploglimit. The first option, as the name implies, can replay the action in Oplog.bson. A second option

Back again to do the introduction. Let's look at an example:

First we simulate a collection of Foo, which has an ongoing insert operation,

Use test  for (var i = 0; i < 100000; i++) {    Db.foo.insert ({a:i});}

The mongodump is then simulated once during the insertion process and the--oplog is specified.

Mongodump-h 127.0.0.1--oplog

Note the--oplog option is only valid for full-library export, so you cannot specify the-D option. Because the entire instance of the change operation will be concentrated in the local library in the Oplog.rs collection.

According to the above, the time system starting from dump will record all the Oplog to Oplog.bson, so we get these files:

Yaoxing ~ $ ll dump/1442470:  OPLOG.BSONDRWXR2 yaoxing yaoxing   4096: Test

Where test is the database we just used, Oplog.bson is all the operations performed during the export. If you are curious about the content in Oplog.bson, you can use the Bsondump tool to view its contents, for example:

{"H": {"$numberLong": "2279811375157953332"}, "ns": "Test.foo", "O": {"_id": {"$oid": "55f834ae6b530b5854f9d6ee"}, "a" : 7784.0}, "Op": "I", "ts": {"$timestamp": {"T": 1442329774, "I": 3248}}, "V": 2}

From the Oplog.bson we select the first and last content to observe.

{"H": {"$numberLong": "2279811375157953332"}, "ns": "Test.foo", "O": {"_id": {"$oid": "55f834ae6b530b5854f9d6ee"},  "A": 7784.0}, "Op": "I", "ts": {"$timestamp": {"T": 1442329774, "I": 3248}}, "V": 2} ... {"H": {"$numberLong": " -1177358680665374097"}, "ns": "Test.foo", "O": {"_id": {"$oid": "55f834b26b530b5854f9fa5e" },"a": 16856.0}, "Op": "I", "ts": {"$timestamp": {"T": 1442329778, "I": 1361}}, "V": 2}

The Scarlet Letter section shows that, from the beginning of the Mongodump, the loop is carried to the i=7784, and the loop is made to i=16856 at the end of the operation. And take a look at the last piece of data in Test/foo.bson.

{"_id": {"$oid": "55f834ae6b530b5854f9d73d"},"a": 7863.0}

It can be found that the final dump of the data is neither the first state nor the final state, but a random state in the middle. This is precisely because the collection is constantly changing.

Then use Mongorestore to recover:

Yaoxing ~ $ mongorestore-h127.0.0.1--Oplogreplay Dump -- the-19T01: A:20.095+0800Building a list of DBS and collections to restore from dumpdir -- the-19T01: A:20.095+0800Reading metadata forTest.foo from -- the-19T01: A:20.096+0800restoring Test.foo from -- the-19T01: A:20.248+0800restoring indexes forcollection Test.foo from metadata2015-09-19t01:22:20.248+0800 finished restoring Test.foo (7864 documents) 2015-09-19t01:22:20.248+0800 replaying Oplog -- the-19T01: A:20.463+0800     Done

Note the two sentences of the Scarlet letter, the first sentence indicates that 7,864 documents were recovered from the Test.foo collection; the second sentence shows that all operations in Oplog are replayed. So theoretically foo should have 16,857 documents (7,864 from Foo.bson and the rest from Oplog.bson). Verify that:

Test:primary> db.foo.count ()16857

This is the real function of the mongodump with Oplog.

1.2.3 from somewhere else, Oplog.

Smart as you might have thought, since the dump out of the data with Oplog can restore the database to a state, it is not to have a backup from a point in time to start the dump data, plus the Oplog after the start of dump, if oplog long enough, Is it possible to restore the database to any subsequent state? Yes! The fact that replica set is dependent on Oplog's replay mechanism is working. When secondary first joined replica set initial sync is the equivalent of doing mongodump, and then only need to constantly synchronize and replay the oplog.rs in the data, the secondary and primary to achieve the purpose of synchronization.

Since Oplog has always been in the oplog.rs, why do we need to specify--oplog at Mongodump? Do you have to take it from the oplog.rs when you need it? The answer is yes, you can just dump the data and do not need to oplog. It can be taken from the oplog.rs when needed. But the premise is that the oplog time window (Forget the time window concept, please forward) must be able to cover the start time of dump.

Understand this truth, theoretically, as long as our mongodump do frequently enough to ensure that the database can be restored to any point in time in the past. A paid backup of MMS (now called Cloud Manager) is also working with this principle. Assuming that the Oplog time window has 24 hours, then theoretically as long as I complete a dump within every 24 hours, you can guarantee the 24-hour point-in-time data recovery after dump. In the Oplog time window is about to slide out 24 hours, as long as the next dump in time to complete, you can have a 24-hour security period.

To do a test. Still use the previous method to simulate a period of database insert operations:

Use test  for (var i = 0; i < 100000; i++) {    Db.foo.insert ({A:I});}

Do mongodump at once without--oplog:

Yaoxing ~/dump $ mongodump-h 127.0.0.1
2015-09-24t00:06:11.929+0800writing test.system.indexes to Dump/test/system.indexes.bson
2015-09-24t00:06:11.929+0800done dumping Test.system.indexes (1 document)
2015-09-24t00:06:11.929+0800writing Test.foo to Dump/test/foo.bson
2015-09-24t00:06:11.963+0800done Dumping Test.foo (11162 documents)

We can see the state of the i=11162 when we export it. After the insert is completed, theoretically our Foo collection should have 100,000 records

> Use testswitched to db Test> db.foo.count ()100000

Now suppose I had a mistake:

>"nremoved": 100000})

Even worse, I then inserted a piece of data into Foo.

> Db.foo.insert ({a:100001"ninserted": 1})

Now how to turn back the clock and return to the pre-disaster state? Since we had a dump before the disaster, it would be nice to save Oplog before the Oplog time window could cover the export time:

127.0. 0.1 -D local- -24t00:41.040+0800    Writing local.oplog.rs to dump/local/oplog.rs.bson- -24t00:: 41.525+0800done     dumping local.oplog.rs (200003 documents)

Why would there be 200,003 records? Please use the Bsondump tool yourself to see what's going on.

As you can see from the front, the database is restored using Mongodump plus Oplog.bson (note the file location). The Dump/local/oplog.rs.bson here is actually the oplog.bson we need. So rename it and put it in the right place, and a simulated recovery environment is ready.

Yaoxing ~/18464118900271 :  OPLOG.BSONDRWXR2 yaoxing yaoxing     4096: Test

But this oplog.bson contains all the disaster operations ah, if the overall recovery, it is tantamount to let the time back, and then see the tragedy repeats again. The heart is broken ... At this point you need a new friend, the--oploglimit mentioned above. When used with--oplogreplay, you can limit the point in time at which playback is replayed. So the important question is how to find the point in time when the disaster happened. is still bsondump. If you are familiar with Linux commands, you can manipulate them directly in the pipeline. If not, then dump to the file, and then open with a text editor slowly find out. What we need to find is "OP": "D", which indicates a delete operation. It can be found that there are 100,000 deletions in the Oplog.bson, which is actually a single piece that deletes the record, which is why the remove ({}) operation is so slow. If you Drop () a collection, it will be much faster, and the actions it makes should be tried by the reader.

Yaoxing ~/dump $ bsondump Oplog.bson |grep "\ "op\": \ "d\""|Head{"b":true,"h":{"$numberLong":"5331406427126597755"},"NS":"Test.foo","o":{"_id":{"$oid":"5602cdf1befd4a4bfb4d149b"}},"op":"D","TS":{"$timestamp": {"T": 1443024507, "I": 1}},"v":2}...

What we need in this record is the Red $timestamp section, which represents the time it takes for this operation to occur, and the time that our--oploglimit need to pass in, but the format changes slightly:

Yaoxing ~ $ mongorestore-h127.0.0.1--oplogreplay--oploglimit"1443024507:1"dump/ -- the-24t00: the:09.533+0800Building a list of DBS and collections to restore from dumpdir -- the-24t00: the:09.534+0800Reading metadata forTest.foo from -- the-24t00: the:09.534+0800restoring Test.foo from -- the-24t00: the:09.659+0800restoring indexes forcollection Test.foo from metadata -- the-24t00: the:09.659+0800Finished restoring Test.foo (11162documents) -- the-24t00: the:09.659+0800replaying Oplog -- the-24t00: the:11.548+0800     Done

1443024507 is "T" in $timestamp and 1 is "I" in $timestamp. After this configuration, Oplog will replay to this point in time, that is, to avoid the first DELETE statement and its subsequent operations, the database stays in pre-disaster state. Verify that:

Rs0:primary> db.foo.count ()100000

1.3 Summary

In conjunction with the knowledge of the technique, we can summarize some of the MongoDB backup guidelines (only for replica or master/slave) to meet these criteria and MongoDB can perform point-in-time recovery operations:

• The time interval for any two data backups (the first backup starts to the end of the second backup) cannot exceed the Oplog time window coverage.
• On the basis of the last data backup, a full Oplog backup is performed before the Oplog time window does not slide out of the point at which the last backup ended. Take full account of the time required for Oplog backup and weigh the server space to determine the Oplog backup interval.

Considerations in Practical Applications:

• Taking into account the Oplog time window is a change value, please pay attention to the time of Oplog time window.
• You must have enough time to dump the required oplog.rs before the Oplog time window slides out of the effective time, please leave enough time, do not back up the full time window.
• When the disaster occurs, the first thing to do is to stop the database write operation, the past Oplog slide out of the time window. In particular, the remove ({}) operation, like the above, inserts a large number of D records in an instant, causing the Oplog to quickly slide out of the time window.

2. External Method 1. Mms/cloud Manager

Formerly known as MMS, the monitoring function is free, the backup function is charged; the first two months changed to Cloud manager and began to charge for monitoring functions, add an Operation Automation robot, very silly, interested can play. However, since the charges, I believe that China's national conditions have determined that few companies will be willing to use. So do not focus on the introduction, as long as it can provide any time point of data recovery can be, powerful, local tyrants company Welcome to use.

2. Disk Snapshots

Common such as LVM snapshots, which can be used to back up MongoDB, but must turn on MONGODB Journal and journal must and data files on a volume. It should be said that most people open journal, right? Without journal, a power outage would lose up to 60s of data.

Snapshot is actually a more mongodump/mongorestore faster backup method, quite practical, but only to provide backup time point of data recovery, the role is limited. Do not expand in detail and are interested in viewing official documents. In theory, snapshots can also be combined with Oplog replay to achieve point-in-time data recovery. But I do not have a personal experiment, interested friends can play and do an introduction.

General backup strategy for MongoDB