Rsync, rsync

Rsync, rsync
Rsync: 1. About rsync

Rsync is short for remote synchronize. It is a remote synchronization software that is useful in projects. Its rsync algorithm (the core algorithm of rsync introduced in rsync's Core Algorithm in cool shell) provides a fast method for client and Remote File Server synchronization.

Rsync features convenient installation and configuration, and maintains symbolic links, hard links, permissions, file time, and file ownership during file synchronization. Security provides password, read/write permission control, and IP address permission control.

2. rsync installation and configuration

I use ubuntu and have pre-installed rsync, so I don't need to install it. My configuration file is/etc/rsyncd. conf. The configuration is as follows. For more information, see rsyncd. config:

read only = yeslist = yesuid = nobodygid = nogrouppid file = /var/run/rsyncd.pidsecrets file = /etc/rsyncd.secretslog file = /var/log/rsync.log[test]comment = Welcomepath = /home/ssj/rsynctesthosts allow = 127.0.0.1 10.211.55.11auth users = rsynctest

Read only sets the synchronization directory read-only. uid and gid are the users and user groups used for synchronization. Pid file and log file are the storage addresses of pid files and log files.

Test Is the synchronization directory, and path is the synchronization directory. Note that the variable is followed by a comment such as # XXX. If it is added, a chroot error is returned. In addition, hosts allow restricts the synchronized ip addresses. Multiple ip addresses are separated by spaces. Auth users and secrets files are authenticated user and password files respectively. The secrets file Permission must be set to 600. Otherwise, an error will occur during synchronization. Format: User name: password. The user name here is not necessarily a system user. you can name it as needed. For example, my account isrsynctest:testpasswd.

3. rsync command

There are two modes for rsync synchronization: one is to connect through a remote shell such as ssh, and the other is to directly connect to the rsync server through TCP. The ssh port needs to be set for synchronization such as ssh, Which is troublesome. Generally, I directly connect to the rsync server. The most common commands for synchronization are as follows:

rsync -avz --progress --delete rsync://rsynctest@10.211.55.11/test/ rsynctarget/

Here, rsynctest is the user name for synchronization and 10.211.55.11 is the ip address of my rsync server. Run the command and enter the password. If you do not want to enter the password every time, you can add the Parameter--password-file=/etc/rsyncd.secretsHere, rsyncd. secrets is the file on your rsync client machine, and the permission is also 600. The content format is different from that on the rsync server. Here you only need the password testpasswd. -Progress displays the synchronization progress.-delete deletes unnecessary files in rsynctarget.

These options avz are commonly used, where v is verbose, z is compression, and A is equivalent to rlptgoD. r is the recursive synchronization directory. If-r is not added,-d is used, only the corresponding directory will be synchronized, but the files under the directory will not be synchronized; l synchronization Symbolic Link, p synchronization file will remain unchanged, t keep the file time, g keep the file group, o keep the user of the file, D keep the device file. In general, you can use avz. In addition, option-P is also commonly used, indicating that some files are synchronized. If the synchronization is interrupted, You can resume the transfer.

Note that when synchronizing a sub-directory, for example, synchronizing rsynctest/testdir to the directory, if the command is as follows:rsync -avz rsync://rsynctest@10.211.55.11/test/testdir rsynctarget/Because "/" is not added after testdir, the synchronization result is that there is a testdir directory under the rsynctarget directory, that is, rsynctarget/testdir. If the command is`rsync -avz rsync://rsynctest@10.211.55.11/test/testdir/ rsynctarget/The result is that all the files under the testdir directory are synchronized to the rsynctarget directory, and there is no testdir directory under the rsynctarget directory.

4. Instance

For convenience, I created a directory for rsync testing. The contents in the rsynctest directory are as follows:

root@ubuntu:/home/ssj/rsynctest# ls -latotal 20drwxr-xr-x  3 nobody nogroup 4096 Mar  7 16:33 .drwxr-xr-x 28 ssj    ssj     4096 Mar  7 16:13 ..-rw-r--r--  2 nobody nogroup   13 Mar  7 16:32 link.txtlrwxrwxrwx  1 nobody nogroup   12 Mar  7 15:13 outer.txt -> ../outer.txtlrwxrwxrwx  1 nobody nogroup    8 Mar  7 14:52 softlink.txt -> test.txtdrwxr-xr-x  2 nobody nogroup 4096 Mar  7 14:51 testdir-rw-r--r--  2 nobody nogroup   13 Mar  7 16:32 test.txt

The hard link of the worker.

The contents of the target directory rsynctarget for synchronization are as follows:

root@ssj-mbp ssj#ls -la rsynctarget/total 8drwxr-xr-x   3 65534  65534   102  3  7 17:15 .drwxr-xr-x@ 94 ssj    staff  3196  3  6 23:38 ..-rw-r--r--   1 root   65534     7  3  7 17:15 myfile.txt

Different parameters are used, and the synchronization results are different.

1) You do not need the-a parameter or the-og parameter. The command is as follows:

rsync -rpvz --progress --delete rsync://rsynctest@10.211.55.11/test/ rsynctarget/ --password-file=/etc/rsyncd.secrets

The slave node is not synchronized. After synchronization, the hard link test.txt is treated as a new file, and the hard link relationship is not retained. If you want to retain it, add the-H option. In addition, the file owner is not the original nobody: nogroup. If you want to retain it, add the-og parameter. After the command is synchronized, the contents of the rsynctarget directory are as follows:

root@ubuntu:/home/ssj# ls -la rsynctarget/total 20drwxr-xr-x  3 root root 4096 Mar  7 17:31 .drwxr-xr-x 29 ssj  ssj  4096 Mar  7 17:31 ..-rw-r--r--  1 root root   13 Mar  7 17:31 link.txtdrwxr-xr-x  2 root root 4096 Mar  7 17:31 testdir-rw-r--r--  1 root root   13 Mar  7 17:31 test.txt

2) use the-a parameter. The command is as follows:

rsync -avz --progress --delete rsync://rsynctest@10.211.55.11/test/ rsynctarget/ --password-file=/etc/rsyncd.secrets

After synchronization, the content of the rsynctarget directory is as follows. After-a is used, the file owner is retained and the soft connection is synchronized, but the hard link relationship is not retained. If you want to retain it, add-H.

root@ubuntu:/home/ssj# ls -la rsynctarget/total 20drwxr-xr-x  3 nobody nogroup 4096 Mar  7 16:33 .drwxr-xr-x 29 ssj    ssj     4096 Mar  7 17:31 ..-rw-r--r--  1 nobody nogroup   13 Mar  7 16:32 link.txtlrwxrwxrwx  1 nobody nogroup   12 Mar  7 15:13 outer.txt -> ../outer.txtlrwxrwxrwx  1 nobody nogroup    8 Mar  7 14:52 softlink.txt -> test.txtdrwxr-xr-x  2 nobody nogroup 4096 Mar  7 14:51 testdir-rw-r--r--  1 nobody nogroup   13 Mar  7 16:32 test.txt

3) Another important parameter is-P. Note that this P is not the meaning of progress. It is the meaning of partial, that is to say, if the synchronization process is interrupted (for example, if the synchronization is stopped or the network problem causes the synchronization to be interrupted), if a file is not completely transferred, whether to retain the transmitted part is controlled by-P. Here, I added a test.tar.gz file under the rsynctestdirectory with a size of 2 M. I used the bwlimit parameter to limit the transmission speed to 10 K/second. The next re-transmission will be resumed, that is, the synchronized part will not be re-transmitted. In the synchronization process, the ghost file name is in the format of .tornado.tar.gz. qesuQR. If a program retrieves and processes data from the synchronization directory, it is best not to use the-P parameter, because if a network problem causes a part of the file to be passed and then processed, this will affect the processing result.

rsync -avzP --progress --delete --bwlimit=10 rsync://rsynctest@10.211.55.11/test/ rsynctarget/ --password-file=/etc/rsyncd.secrets

5. References

• Rsync Security Configuration
• Rsync Core Algorithm
• Rsync remote synchronization of backup data