Linux rsync Command detailed
| [Date: 2011-04-28] |
Source: Linux Community Author: Linux |
1. What is rsync
Rsync (remotesynchronize) is a remote data synchronization tool that allows you to quickly synchronize files between multiple hosts by Lan/wan. Rsync uses the so-called "rsync algorithm" to synchronize files between local and remote two hosts, and this algorithm transmits only a different portion of two files, rather than sending it all at a time, so it's very fast.
Rsync was originally used as a tool to replace RCP and is currently maintained by rsync.samba.org, so the format of rsync.conf files is similar to Samba's main configuration file. Rsync can be used through rsh or SSH, and it can run in daemon mode, and Rsyncserver will open a 873 port to wait for the client to connect when running daemon. When you connect, Rsyncserver checks to see if the passwords match, and if you check by password, you can begin file transfer. When the first connection completes, the entire file is transmitted once, and then only incremental backups are made.
Rsync supports most Unix-like systems, both Linux, Solaris, and BSD are well tested. In addition, it also has a corresponding version under the Windows platform, such as Cwrsync and Sync2nas tools.
The basic features of rsync are as follows:
1. Can be mirrored to save the entire directory tree and file system;
2. It is easy to maintain the original file permissions, time, soft and hard links;
3. Installation without special permission;
4. Optimized flow, high efficiency of file transfer;
5. Can use Rsh, ssh and other means to transfer files, of course, can also be connected through the direct socket;
6. Support for anonymous transmission.
2, rsync synchronization algorithm
Rsync only synchronizes files very quickly because the Rsync synchronization algorithm calculates the data that needs to be backed up in a very short time, and the synchronization algorithm for Rsync is described as follows:
Assume that similar files A and B are synchronized between two computers, 1th and 2nd, where 1th has access to file A and 2nd has access to file B. and assume that the network bandwidth between host 1th and 2nd is very small. Then the rsync algorithm will be done with the following five steps:
1, 2nd to split file B into a set of not overlapping blocks of fixed size S byte, the last piece may be smaller than S.
2, 2nd for each piece of segmented data to perform two checks: one is a 32-bit rolling weak checksum, the other is a 128-bit MD4 strong checksum
3, 2nd to send these results to 1th.
4, 1th by searching for file A of all size S of the block (offset can be optional, not necessarily a multiple of s), to find a piece of file B with the same weak check code and strong check code of the data block. This work can be done quickly with the help of scrolling checksums.
5, 1th sent to 2nd a series of instructions to generate file A on the 2nd number of the backup. Each instruction here is either proof that file B has a block of data without retransmission, or a block of data that must not match any of the data blocks of file B.
3, rsync parameter description
3.1 rsyncd.conf configuration file
-, global parameters
All parameters before [module] in a file are global, and of course you can define a module parameter in the Global Parameters section, at which point the value of the parameter is the default value for all modules.
Port
Specifies the port number used by the daemon, which defaults to 873.
MOTD file
The MOTD file parameter is used to specify a message file that, when the client connects to the server, displays the contents of the file to the customer, and there is no MOTD file by default.
Log file
"Log File" specifies the log files for rsync and does not send the log to syslog. For example, you can specify "/var/log/rsyncd.log".
PID File
Specifies the PID file for rsync, which is usually specified as "/var/run/rsyncd.pid."
Syslog facility
Specifies the message level for rsync when sending log messages to syslog, and common message levels are: Uth, Authpriv, cron, daemon, Ftp,kern, lpr, mail, news, security, Sys-log, user, UUCP , Local0, Local1,local2, Local3,local4, LOCAL5, Local6 and LOCAL7. The default value is daemon.
Second, module parameters
The main is to define which directory the server is to be synchronized. The format must be in the form of "[Module]", which is the name that is seen on the rsync client, in fact a bit like the share name provided by the Samba server. The data that the server really synchronizes is specified by path. We can specify multiple modules according to our own needs, and the following parameters can be defined in the module:
Comment
Assign a description to the module, which is displayed to the customer, along with the module name, when the client connects to the module list. The default does not describe the definition.
Path
Specifies the directory tree path for the module to be backed up, which must be specified.
Use Chroot
If "Usechroot" is specified as true, rsync first chroot to the directory specified by the path parameter before transferring the file. The reason for this is to implement additional security protection, but the disadvantage is that you need to roots permissions, and you cannot back up the directory files that point to external symbolic connections. The chroot value is true by default.
Uid
This option specifies the UID that the daemon should have when the module transmits files, with the GID option used to determine which file permissions to access, and the default value is "nobody".
Gid
This option specifies the GID that the daemon should have when the module transmits files. The default value is "nobody".
Max connections
Specify the maximum number of concurrent connections to the module to protect the server, and a connection request exceeding the limit will be told to try again later. The default value is 0, which means there is no limit.
List
This option sets whether the module should be listed when the client requests a list of modules that can be used. If you set this option to False, you can create a hidden module. The default value is true.
Read Only
This option sets whether the customer is allowed to upload files. If true, any upload requests will fail, and if it is false and the server directory read-write permission is allowed, the upload is allowed. The default value is true.
Exclude
Use to specify multiple files or directories (relative paths) separated by spaces and add them to the exclude list. This equates to specifying a pattern using--exclude in the client command, and a module can specify only one exclude option. However, it is important to note that this option has a certain security problem, the customer is likely to bypass the exclude list, if you want to ensure that a particular file can not be accessed, it is best to combine the uid/gid option.
Exclude from
Specifies a filename that contains the definition of the Exclude mode from which the server reads the Exclude list definition.
Include
Used to specify that files or directories that meet the requirements are not excluded. This equates to specifying patterns using--include in client commands, combining include and exclude to define complex exclude/include rules.
Include from
Specifies a file name that contains the definition of the include mode, from which the server reads the Include list definition.
Auth Users
This option specifies the list of user names separated by a space or comma, only those users who are allowed to connect to the module. There is no relationship between the user and the system user here. If "Auth users" is set up, then the client sends a connection request to the module and the challenged is requested by rsync to authenticate the Challenge/response authentication protocol used here. The user's name and password are stored in plaintext in the file specified by the Secrets file option. The module can be connected by default without a password (that is, anonymous).
Secrets file
This option specifies a file that contains a defined user name: a password pair. This file is only useful when "auth users" is defined. The file contains a username:passwd pair per line. Generally speaking, passwords are best not to exceed 8 characters. There is no default secures file name, and a limited designation is required (for example:/ETC/RSYNCD.PASSWD). Note: The permissions for this file must be 600 or the client will not be able to connect to the server.
Strict modes
This option specifies whether to monitor the permissions of the password file, and if true, the password file can only be accessed by a user who is running the identity of the rsync server, and no other user can access the file. The default value is true.
Hosts allow
This option specifies which IP clients are allowed to connect to the module. The customer mode definition can be in the following form:
A single IP address, for example: 192.167.0.1
The entire network segment, for example: 192.168.0.0/24, can also be 192.168.0.0/255.255.255.0
multiple IP or network segments need to be separated by spaces, and "*" means all, by default, all host connections are allowed.
Hosts Deny
Specifies that machines that are not allowed to connect to the Rsync server can be defined by using the hosts allow definition. The default is no hosts deny definition.
Ignore errors
Specifies that RSYNCD ignores IO errors on the server when determining whether to run a transfer, and generally rsync skips--delete operations when an IO error occurs to prevent serious problems caused by temporary resource shortages or other IO errors.
Ignore nonreadable
Specifies that the RYSNC server completely ignores files that the user does not have access to. This is useful for situations where files that need to be backed up should not be available to a backup person.
Lock file
Specifies the lock file that supports the Max connections parameter, and the default value is/var/run/rsyncd.lock.
Transfer logging
Enables RSYNC servers to use FTP-formatted files to record download and upload operations in their own separate logs.
Log format
This option allows users to customize the log file's fields using transfer logging. The format is a string that contains a format definition character, and the format definition you can use is as follows:
%H Remote Host Name
%a Remote IP Address
%l file length character count
%p Process ID for this rsync session
%o Operation type: "Send" or "recv"
%f file name
%P Module Path
%M Module Name
%t Current Time
%u authenticated username (null when anonymous)
%b The actual number of bytes transferred
%c when the file is sent, the field records the checksum code for the file
The default log format is: "%o%h [%a]%m (%u)%f%l", in general, "%t [%p]" is added to the header of each line. A Perl script called Rsyncstats is also published in the source code to count the log files in this format.
Timeout
This option allows you to override the IP timeout specified by the customer. This option ensures that the rsync server will not wait forever for a crashed client. The timeout unit is seconds, and 0 indicates no time-out definition, which is also the default value. An ideal number for anonymous rsync servers is 600.
Refuse options
This option allows you to define a list of command arguments that are not allowed to be used by the customer for the module. You must use the full name of the command, not the abbreviation. However, when a command is rejected, the server reports an error message and exits. If you want to prevent the use of compression, it should be: "Dont compress = *".
Dont compress
Used to specify files that are not compressed for transfer, and the default is *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2
3.2 rsync Command
After the rsync server configuration is over, the next step is to issue an rsync command on the client to enable the server-side files to be backed up to the client. Rsync is a very powerful tool, and its commands have many feature options, and we'll analyze all of its options below.
The command format for rsync can be in the following six ways:
rsync [OPTION] ... SRC DEST
rsync [OPTION] ... SRC [User@]host:dest
rsync [OPTION] ... [user@] HOST:SRC DEST
rsync [OPTION] ... [user@] HOST::SRC DEST
rsync [OPTION] ... SRC [user@]host::D EST
rsync [OPTION] ... rsync://[user@]host[:P ort]/src [DEST]
