How MongoDB implements Backup compression

Background and principle

Backup of the database is the last barrier to disaster recovery, no matter what type of database you need to set up database backup, MongoDB is no exception. After MongoDB 3.0, the database can be used Wiredtiger storage engine (3.2 version default), in this environment through Mongodump backup, the resulting backup file is much larger than the size of the data storage file. In addition, the general MongoDB store a large amount of data, backup files are relatively large, taking up a lot of disk space. Therefore, it is necessary to study how to implement MongoDB backup compression.

is to execute the command db.stats () to view the information for a database.

The size of the backup file is generally datasize, so we want to compress the backup to reach storagesize or smaller.

a backup idea is to back up first, then compress the backup files . Previously, we used this approach, such as the main compression command as follows

TAR-CF-${targetpath}/${nowtime} | Pigz-p > ${targetpath}/${nowtime}.tgz

(Command explanation: Targetpath}/${nowtime is the backup file to be compressed; Pigz is a Linux compression artifact that can be compressed in parallel;-P is the number of cores for the specified CPU.) ）

However, in this way, it is easy to generate disk performance pressure and space pressure during the process of generating backup files. For one of our servers using a backup-after-compression method, the resulting disk free space changes.

The real hope is to compress at the same time as the backup, so that the available space is more stable . A compressed backup was introduced in MongoDB 3.2 "This MongoDB version must be no less than 3.2". You can use gzip to compress. This is achieved by introducing a new command line option "--gzip" in Mongodump and Mongorestore.

Compression can be used for backups created under the catalog and archive model, and compression also reduces disk space usage.

Test

Test environment:

Test server	Test database	Port	File path
172.x.x.245	Fully prepared instance	17219	/data/mongodb_back
172.x.x.246	Qq_dingding	17218	/data/mongodb_back/qq_dingding

command for Step 1 compression backup:

./mongodump--host 172.x.x.245--port 17219-u user name-P "password"--gzip--authenticationdatabase "admin"--out/data/mongodb_bac K

Size of the file after backup, 97M

At this point, the format of viewing the backup file becomes. gz format

Step 2 Copy the backup file to the remote machine for restore:

The following command is to be in 172. x.x.246, the requirement is to copy files from x.245 to local

scp-r [Email protected]:/data/mongodb_back/qq_dingding

Step 3 to perform the RESTORE command

Commands to execute

./mongorestore--host 172.x.x.246--port 17218-d qq_dingding-u user name-P "password"--gzip--authenticationdatabase "admin"/data /mongodb_back/qq_dingding

After the restore, log in to MongoDB, execute show DBS, and view the data size at 500M.

Additional Information

(1) If you do not use a compressed backup, how big is the file after backup? Backup command:

./mongodump--host 172.x.x.245--port 17219-u user name-P "password"--authenticationdatabase "admin"--out/data/mongodb_back2

View the file size--1.5g after this method is backed up.

Taking this qq_dingding database as an example, the compression rate is (the ratio of the size of the file after compression to the size before compression): 97m/1.5g=97/1536=6.3%

(2) Does this way of compressing backups bring some drawbacks: for example, backup time growth ? (Increased recovery time?) , please self-test, Hee @@@)

Example of an archive backup repository instance (storagesize 150g,datasize 600G)

It takes 1 hours and 55 minutes to compress after first backup.

Compressed backup (specifying--gzip parameters) takes 2 hours and 33 minutes

The resulting backup file size is basically equal, and the backup file produced by the compressed backup method is slightly smaller

Therefore, a compressed backup can cause the backup time to grow.

but from a space use standpoint, we still recommend that you use compressed backup, which has a very high compression ratio (compression ratio of the test case is 6.3%).

How MongoDB implements Backup compression