Rsync takes up a lot of memory when backing up a mass file _ server other

Most Linux distributions have their own rsync, but lower versions, typically 2.6.X
In the 2. In the version of X, rsync backups are list and then backed up (added or deleted), which consumes a lot of memory when working with a large number of files.
At the time of the backup, rsync scanned every file (same directory), in its list of approximately 100 bytes of memory, if the--delete parameter, the memory will be more occupied.
For example, I have a server here, about 8 million of the picture, and update more frequently, the number of files to grow faster, almost 100,000 daily increase. At the time of backup, rsync consumes approximately 2G of memory, consumes a large amount of memory, creates a lack of physical memory on the server, uses swap, and then generates higher iowait (swap memory), which in turn leads to a slower list of rsync and affects the business on the server.
In this case, before the Rsync 3.X appeared, the general suggestion was to split the backup operation into several small backup operations. For example, there are 10 image catalogs backed up together, now split into 10 backup operations, only one backup at a time. In addition, it is recommended that you reduce the depth of the directory, which reduces the number of directories and can reduce the memory consumed by rsync. There is also a software called Digisync, which is specifically used to back up the G-level number of files.

Rsync 3.X uses the incremental file list, which is now backed up (added or deleted) by the side of the list compared to the original 2.X. This saves a lot of time for the backup operation of a large number of files.
As measured, the amount of memory used by Rsync 3.0.4 backup was approximately 4M, almost as much as an Apache process.

The homepage of rsync in http://samba.anu.edu.au/rsync/is currently the latest stable version of rsync-3.1.1

Copy Code code as follows:

cd/usr/src/
wget http://samba.anu.edu.au/ftp/rsync/src/rsync-3.1.1.tar.gz
Tar xzvf rsync-3.1.1.tar.gz
CD rsync-3.1.1
./configure--PREFIX=/USR
Make
Make install

Then run rsync--version to see the version number

It is important to note that both the source host and the destination host must be upgraded to rsync 3.X to use the new features of Rsync 3.X.

It is worth mentioning that since the release of version 2.6.9 on November 6, 2006, 3.0 was released until March 08, making it a long time before people had to find various ways to handle a large number of file backup operations