Rsync incremental transfer large file optimization techniques

Rsync Server Configuration

Make sure you have Rsync installed

Configure/etc/rsyncd.conf, typically installed rsync does not automatically create RSYNCD.CONF, configured as follows

The code is as follows

Copy Code

# Rsync configuration file Secrets file =/etc/rsyncd.secrets #认证用户名和密码文件的名称和位置 MOTD file =/etc/rsyncd.motd #欢迎文件, can edit yourself Read>list = yes UID = root GID = root Use chroot = no Max connections = 5 Log file =/var/log/rsyncd.log PID file =/var/run/rsyncd.pid Lock file =/var/run/rsync.lock # Remote Sync Configuration module [Testsync] Comment = Testsync Directory Path =/home/chijianqiang Auth users = rsync

Generate Password file

The code is as follows	Copy Code
echo "Rsync:password" >>/etc/rsyncd.secrets chmod 600/etc/rsyncd.secrets

Start rsync

The code is as follows	Copy Code
Rsync--daemon--config=/etc/rsyncd.conf

Synchronizing files from the client

The code is as follows	Copy Code
Rsync-avz--delete/clientfilepath Username@192.168.xx.xx:port::modelname--password-file=/etc/rsyncd.secrets RSYNC-VZRTOPG--progress RSYNC@10.11.3.3::TESTSYNC/HOME/CHJQ RSYNC-VZRTOPG--delete--progress RSYNC@10.11.3.3::TESTSYNC/HOME/CHJQ

The addition of the--delete parameter will remove the files from the original CHJQ directory to keep the client and server file systems in full agreement, with caution

Rsync incremental transfer large file optimization techniques

Problem

Rsync is useful for synchronizing data, especially for incremental synchronizations. But there is a situation where it is not very useful to add a particular parameter. For example, you have to sync more than dozens of G files, then the network suddenly disconnected, then you restart the incremental sync. But found that wait for a long time did not carry out data transmission, but the machine IO has been high.
Reason

Rsync specific incremental synchronization algorithm is not very clear. Depending on its performance, it is possible to verify that the transferred portion of the data already exists in a file that is already in an incremental synchronization, and that the checksum is complete before continuing to incrementally synchronize the remaining data in the file. So if the incremental synchronization of a large file with such an algorithm is time-consuming and takes up IO resources.
Method

Spent one hours in the middle of the night looking at Rsync's documentation and found a parameter that quickly restores the incremental sync of large files, –append. Setting the –append parameter calculates the file size and appends the new data directly to the file during an incremental synchronization, which saves the cost IO checksum process. However, it is best to use a security, such as a backup file, only when the source and target files are not changed.