Rsync configuration command details

Rsync configuration commands 1. What is Rsync-Rsync Configuration Parameter Details Rsync (remote synchronize) is a remote data synchronization tool that allows you to quickly synchronize files between multiple hosts through the LAN/WAN. Rsync uses the so-called "Rsync algorithm" to synchronize files between the local and remote hosts. This algorithm only transfers different parts of the two files, instead of transmitting the entire file each time, therefore, the speed is quite fast. Rsync is a tool used to replace rcp. It is currently maintained by rsync.samba.org, so the format of rsync. conf file is similar to the main configuration file of samba. Rsync can be used through rsh or ssh or run in daemon mode. When running in daemon mode, Rsync server opens a port 873 and waits for the client to connect. During connection, the Rsync server checks whether the Password Matches. If the password is checked, file transmission can be started. When the first connection is complete, the entire file will be transferred once, and later only incremental backup is required. Rsync supports most Unix-like systems, including Linux, Solaris, and BSD. In addition, it also has corresponding versions on the windows platform, such as cwRsync and Sync2NAS tools. The basic features of Rsync are as follows: 1. images can be used to save the entire directory tree and file system; 2. it is easy to maintain the permissions, time, and soft and hard links of the original file. 3. installation without special permissions; 4. optimized process, high file transmission efficiency; 5. you can use rsh, ssh, and other methods to transmit files. Of course, you can also use a direct socket connection; 6. supports anonymous transmission. 2. The Rsync synchronization algorithm Rsync only synchronizes files quite fast because the "Rsync synchronization algorithm" can calculate the data to be backed up in a short time, the Rsync synchronization algorithm is described as follows: Assume that files A and B are similar to files A and B synchronized between computers 1 and 2, where 1 has access to file, 2 has access to file B. In addition, it is assumed that the network bandwidth between host 1 and host 2 is very small. The rsync algorithm completes the following five steps: 1 and 2 split file B into a group of data blocks with a fixed size of S that do not overlap, the last part may be smaller than S. 2 and 2 perform two types of verification for each split data block: one is a 32-bit rolling weak verification, and the other is a 128-bit MD4 strong verification. 3 and 2 send the verification results to 1. 4. Search for all data blocks whose size is S in file A on the first day (the offset is optional, not necessarily A multiple of S ), to find data blocks with the same weak verification code and strong verification code as one of file B. This work can be quickly completed using the features of rolling verification. 5. Issue A string of commands on the first and second to generate A backup of file A on the second. Each instruction here is either a proof that file B has a data block and does not need to be re-transmitted, or a data block, this data block must not match any data block of file B. 3. Rsync parameter description 3.1 rsyncd. in the conf configuration file-and global parameters in the file, all parameters before [module] are global parameters. Of course, you can also define the module parameters in the global parameters section, at this time, the value of this parameter is the default value of all modules. Port: Specifies the port number used by the background program. The default value is 873. The motd file "motd file" parameter is used to specify a message file. When the client connects to the server, the content of this file is displayed to the client. By default, there is no motd file. Log file "log file" specifies the log file of rsync, instead of sending the log to syslog. For example, you can specify "/var/log/rsyncd. log ". Pid file specifies the pid file of rsync, which is usually "/var/run/rsyncd. pid ". Syslog facility specifies the message level when rsync sends log messages to syslog. Common Message levels are: uth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, sys-log, user, uucp, local0, local1, local2, local3, local4, local5, local6, and local7. The default value is daemon. 2. Module parameters mainly define the directory on which the server is to be synchronized. The format must be "[module]". This name is the name seen on the rsync client. It is actually a bit like the sharing name provided by the Samba server. The data actually synchronized by the server is specified through path. You can specify multiple modules as needed. The module can define the following parameter: comment specifies a description for the module, this description is displayed to the customer when the customer connects to the module list. No description is defined by default. Path specifies the directory tree path for backup of this module. This parameter must be specified. Use chroot if "use chroot" is specified as true, rsync first chroot to the directory specified by the path parameter before transferring the file. The reason for doing so is to implement additional security protection, but the missing point is that you need to use the root privileges, and cannot back up the directory files pointed to by the external symbolic connection. The default chroot value is true. Uid this option specifies the uid that the daemon should have when the module transfers files. With the gid option, you can determine which file permissions can be accessed. The default value is "nobody ". Gid this option specifies the gid that the daemon should have when the module transfers files. The default value is "nobody ". Max connections specifies the maximum number of concurrent connections for this module to protect the server. connection requests that exceed the limit will be notified and then try again. The default value is 0, that is, there is no limit. List this option sets whether the module should be listed when the customer requests a list of available modules. If this option is set to false, you can create a hidden module. The default value is true. Read only this option sets whether the client is allowed to upload files. If this parameter is set to true, all upload requests will fail. If this parameter is set to false and the server directory read/write permission permits upload. The default value is true. Exclude is used to specify multiple files or directories (relative paths) separated by spaces and add them to the exclude list. This is equivalent to using-exclude in client commands to specify the mode. A module can only specify one exclude option. However, you must note that this option has certain security issues. The customer may bypass the exclude list. If you want to ensure that a specific file cannot be accessed, then it is best to use the uid/gid option together. Exclude from specifies a file name that contains the definition of the exclude mode. The server reads the definition of the exclude list from this file. Include is used to specify files or directories that do not comply with requirements. This is equivalent to using-include in client commands to specify the mode. You can use include and exclude to define complex exclude/include rules. Include from specifies a file name that contains the definition of the include mode. The server reads the definition of the include list from this file. Auth users this option specifies a list of usernames separated by spaces or commas. Only these users can connect to this module. The user here has nothing to do with the System user. If "auth users" is set, the connection request sent by the client to this module will be sent by rsync to challenged for authentication. The challenge/response authentication protocol is used here. The user's name and password are stored in plaintext in the file specified by the "secrets file" option. By default, the module can be connected without a password (that is, the anonymous mode ). Secrets file this option specifies a file that contains a user name: Password pair. This file works only when "auth users" is defined. Each row of the file contains a username: passwd pair. Generally, the password should not exceed 8 characters. The default secures file name does not exist. You must specify a limit (for example,/etc/rsyncd. passwd ). Note: The permission for this file must be 600, otherwise the client will not be able to connect to the server. Strict modes this option specifies whether to monitor the permissions of the password file. If this option is set to true, the password file can only be accessed by users who run the rsync server identity, other users cannot access this file. The default value is true. Hosts allow this option specifies which IP addresses are allowed to connect to the module. The customer mode can be defined in the following format: A single IP address, for example, the entire network segment 192.167.0.1, for example, 192.168.0.0/24, or 192.168.0.0/255.255.255.0 multiple IP addresses or network segments, which must be separated by spaces, "*" indicates all. By default, all hosts are allowed to connect. Hosts deny specifies a machine that is not allowed to connect to the rsync server. You can use hosts allow to define the host. Hosts deny is not defined by default. Ignore errors specifies that rsyncd ignores the IO errors on the server when determining whether to delete the operation during transmission. Generally, rsync will skip the-delete operation when an IO error occurs, to prevent serious problems caused by temporary lack of resources or other IO errors. Ignore nonreadable specifies that the rysnc server completely ignores files that users do not have access. This makes sense when some files in the directory to be backed up should not be backed up by the backup owner. Lock file specifies the lock file that supports the max connections parameter. The default value is/var/run/rsyncd. lock. Transfer logging enables rsync servers to use ftp files to record download and upload operations in their own separate logs. With this option, you can use transfer logging to customize log File fields. The format is a string containing the format specifiers. The format specifiers can be used as follows: % h remote host name % a remote IP address % l file length characters % p process id % o operation type of the rsync session: "send" or "recv" % f file name % P module PATH % m Module name % t current time % u authenticated user name (null when anonymous) % B actual number of bytes transmitted % c when a file is sent, this field records the file's verification code. The default log format is: "% o % h [% a] % m (% u) % f % l". In general, "% t [% p]" is added to the header of each line. In the source code, a perl script program named rsyncstats is released to collect statistics on log files in this format. Timeout overwrites the specified IP timeout value. This option ensures that the rsync server never waits for a crashed client. Timeout is measured in seconds. 0 indicates that no timeout is specified. This is also the default value. An ideal number for anonymous rsync servers is 600. Refuse options allows you to define a list of command parameters that cannot be used by customers for this module. The full name of the command must be used. However, when a command is rejected, the server reports an error message and then exits. To prevent compression, it should be: "dont compress = *". Dont compress is used to specify files that are not compressed and then transmitted. The default value is *. gz *. tgz *. zip *. z *. rpm *. deb *. iso *. bz2 *. after the Rsync command of tbz 3.2 is configured on the rsync server, the next step is to issue the rsync command on the client to back up the files on the server to the client. Rsync is a very powerful tool, and its commands have many special options. The following describes the options one by one. The Command Format of Rsync can be as follows: rsync [OPTION]... src dest rsync [OPTION]... SRC [USER @] HOST: DEST rsync [OPTION]... [USER @] HOST: src dest rsync [OPTION]... [USER @] HOST: src dest rsync [OPTION]... SRC [USER @] HOST: DEST rsync [OPTION]... rsync: // [USER @] HOST [: PORT]/SRC [DEST] corresponds to the preceding six command formats. rsync has six different working modes: 1) copy a local file. This mode is enabled when the SRC and DES paths do not contain a single colon ":" separator. For example, rsync-a/data/backup 2) uses a remote shell program (such as rsh and ssh) to copy the content of the local machine to the remote machine. This mode is enabled when the DST path address contains a single colon ":" separator. For example, rsync-avz *. c foo: src 3) uses a remote shell program (such as rsh and ssh) to copy the contents of the remote machine to the local machine. This mode is enabled when the SRC address path contains a single colon ":" separator. For example, rsync-avz foo: src/bar/data 4) copy files from the remote rsync server to the local machine. This mode is enabled when the SRC path information contains the separator. For example: rsync-av root@172.16.78.192: www/databack 5) copy files from a local machine to a remote rsync server. This mode is enabled when the DST path information includes the separator. For example: rsync-av/databack root@172.16.78.192: www 6) List of remote machine files. This is similar to rsync transmission, but you only need to omit the local machine information in the command. For example, rsync-v rsync: // 172.16.78.192/www rsync:-v, -- verbose mode output-q, -- quiet simplified output mode-c, -- checksum: Enable the checksum function to force file transfer verification-a, -- archive mode, indicating that the file is transmitted recursively and all file attributes are kept, which is equal to-rlptgoD-r, -- recursive processes-R in recursive mode for subdirectories, -- relative uses relative path information-B, -- backup to create a backup, that is, when the target already has the same file name, rename the old file ~ Filename. You can use the -- suffix option to specify different backup file prefixes. -- Backup-dir: backs up files (for example ~ Filename) is stored in the directory. -Suffix = SUFFIX defines the backup file prefix-u. -- update only performs updates, that is, skipping all files that already exist in DST and whose file time is later than the backup time. (Do not overwrite the updated file)-l, -- links retains the soft link-L, -- copy-links: process soft links like regular files -- copy-unsafe-links: only copies links other than the SRC path directory tree -- safe-links ignores the directory tree pointing to the SRC path other links-H, -- hard-links keep hard links-p, -- perms keep File permissions-o, -- owner keep file owner information-g, -- group keep file group information-D, -- devices: Keep Device File Information-t, -- times: Keep file time information-S, -- sparse performs special processing on sparse files to save DST space-n, -- dry-run which files will be transmitted-W, -- whole-file to copy files without incremental detection-x, -- one-file-system: Do not span the file system boundary-B. -- block-size = SIZE indicates the block size used by the algorithm. The default value is Is 700 bytes-e, -- rsh = COMMAND specifies to use rsh, ssh for Data Synchronization -- rsync-path = PATH specifies the path of the rsync COMMAND on the remote server-C, -- cvs-exclude automatically ignores files in the same way as CVS to exclude files that do not want to be transmitted. -- existing only updates files that already exist in DST, instead of backing up those new files -- delete those files that are not in the dst src -- delete-excluded also delete the files that are excluded by this option at the receiving end -- after the delete-after transmission ends delete -- ignore-errors and delete it when I/o errors occur in a timely manner -- max-delete = NUM delete a maximum of NUM files -- partial keep those files that are not completely transmitted for any reason, to accelerate the subsequent re-transmission -- force Delete directory, even if it is not empty -- numeric-ids does not set the number user and group ID It is configured with the user name and group name -- timeout = time ip timeout TIME, in seconds-I, -- ignore-times does not skip files with the same time and length -- size-only. when determining whether to back up a file, only check the file size, regardless of the file time -- modify-window = NUM determines whether the file is time-stamped, the default value is 0-T -- temp-dir = DIR. Create a temporary file in DIR -- compare-dest = DIR and compare the files in DIR to determine whether to back up-P is equivalent to -- partial -- progress display. backup process-z, -- compress compresses the backup files during transmission -- exclude = PATTERN specifies to exclude the file mode that does not need to be transmitted -- include = PATTERN specifies the file mode to be transmitted without exclusion -- exclude- from = FILE exclude files in the specified mode -- include-from = FILE does not exclude files that match the specified pattern -- version printing version information -- address is bound to a specific address -- config = FILE to specify other configuration files. The default rsyncd is not used. conf file -- port = PORT specify other rsync service ports -- blocking-io for remote shell use blocking IO-stats to give the transmission status of some files -- progress actual transmission process -- log -format = formAT specify the log file format -- password-FILE = FILE get the password from the file -- bwlimit = KBPS limit I/O bandwidth, KBytes per second-h, -- help display help information 4. Rsync uses instance 4.1 and SSH Mode 1. First, start the ssh service on the server: # service sshd start to start sshd: [OK] 2. Use rsync for synchronization. The user uses the rsync command to back up the data on the server, SSH mode is through the System user to back up, as follows: # rsync-vzrtopg -- progress-e ssh -- delete work@172.16.78.192: /www/*/databack/experiment/rsyncwork@172.16.78.192's password: Refreshing file list... 5 files to considertest/a 0 100% 0.00kB/s 527: 35: 41 (1, 20.0% of 5) B 67 100% 65.43kB/s 0:00:00 (2, 40.0% of 5) c 0 100% 0.00kB/s 527: 35: 41 (3, 60.0% of 5) dd 100663296 100% 42.22 MB/s 0:00:02 (4, 80.0% of 5) Sent 96 bytes provisioned ed 98190 bytes 11563.06 bytes/sectotal size is 100663363 speedup is 1024.19 the information above describes the entire backup process and the total size of the backup data. 4.2 Background Service Mode 1. Start rsync service edit/etc/xinetd. d/rsync file, change disable = yes to disable = no, and restart the xinetd service as follows: # vi/etc/xinetd. d/rsync # default: off # description: The rsync server is a good addition to an ftp server, as it \ # allows crc checksumming etc. service rsync {disable = no socket_type = stream wait = no user = root server =/usr/bin/rsync server_args = -- daemon log_on_failure + = USERID} #/etc/init. d/xinetd r Estart: Stop xinetd: [OK] Start xinetd: [OK] 2. After the rsync program is installed by default in the configuration file is created, the main configuration file of rsync is not automatically created and needs to be created manually, the main configuration file is "/etc/rsyncd. conf, create the file and insert the following content: # vi/etc/rsyncd. confuid = rootgid = rootmax connections = 4log file =/var/log/rsyncd. logpid file =/var/run/rsyncd. pidlock file =/var/run/rsyncd. locksecrets file =/etc/rsyncd. passwdhosts deny = 172.16.78.0/22 [www] comment = backup webpath =/wwwread only = noexclude = testauth users = work3, You cannot use the System user to authenticate the client by creating a password file. Therefore, you must create a password file in the format of "username: password". the user name and password can be defined at will, it is better not to be consistent with the system account, and set the permission of the created password file to 600, which is described in detail in the previous module parameters. # Echo "work: abc123">/etc/rsyncd. passwd # chmod 600/etc/rsyncd. passwd 4, backup to complete the above work, now you can back up the data, as follows: # rsync-avz -- progress -- delete work@172.16.78.192: www/databack/experiment/rsyncPassword: refreshing ing file list... 6 files to consider. /files... a 0 100% 0.00kB/s 528: 20: 41 (1, 50.0% of 6) B 67 100% 65.43kB/s 0:00:00 (2, 66.7% of 6) c 0 100% 0.00kB/s 528: 20: 41 (3, 83.3% of 6) dd 100663296 100% 37.49 MB/s 0:00:02 (4,100.0% of 6) sent 172 bytes provisioned ed 98276 bytes 17899.64 bytes/sectotal size is 150995011 speedup is 1533.755. When server data fails, in this case, the server needs to be restored through the client data, provided that the server allows the client to have the write permission; otherwise, the server cannot be recovered directly from the client, use rsync to restore data as follows: # rsync-avz -- progress/databack/experiment/rsync/work@172.16.78.192: wwwPassword: building file list... 6 files to consider. /AB 67 100% 0.00kB/s 0:00:00 (2, 66.7% of 6) c Sent 258 bytes converted ed 76 bytes 95.43 bytes/sectotal size is 150995011 speedup is 452080.875. The sample scripts here are examples on the rsync Website: 1. Perform Incremental backup of data to the central server every seven days #! /Bin/sh # This script does personal backups to a rsync backup server. you will end up # with a 7 day rotating incremental backup. the incrementals will go # into subdirectories named after the day of the week, and the current # full backup goes into a directory called "current" # tridge@linuxcare.com # directory to backupBDIR =/home/$ USER # excludes file-this contains a wildcard pattern per line of fi Les to excludeEXCLUDES = $ HOME/cron/excludes # the name of the backup machineBSERVER = owl # your password on the backup serverexport RSYNC_PASSWORD = XXXXXX ########### ######################################## #################### BACKUPDIR = 'date + % a' OPTS = "-- force -- ignore-errors -- delete -excluded -- exclude-from = $ EXCLUDES -- delete -- backup-dir =/$ BACKUPDIR-a "export PATH = $ PATH: /bin:/usr /Local/bin # the following line clears the last weeks incremental directory [-d $ HOME/emptydir] | mkdir $ HOME/emptydirrsync -- delete-a $ HOME/emptydir/$ BSERVER:: $ USER/$ BACKUPDIR/rmdir $ HOME/emptydir # now the actual transferrsync $ OPTS $ BDIR $ BSERVER: $ USER/current2. Back up to an idle hard disk #! /Bin/sh export PATH =/usr/local/bin:/usr/bin:/bin LIST = "rootfs usr data data2" for d in $ LIST; domount/backup/$ drsync-ax -- exclude fstab -- delete/$ d // backup/$ d/umount/backup/$ ddone DAY = 'date "+ % A" 'rsync -a -- delete/usr/local/apache/data2/backups/$ DAYrsync-a -- delete/data/solid/data2/backups/$ DAY3: mirroring the cvs tree of vger.rutgers.edu #! /Bin/bash cd/var/www/cvs/vger/PATH =/usr/local/bin:/usr/freeware/bin:/usr/bin: /bin RUN = 'lps x | grep rsync | grep-v grep | wc-l 'if ["$ RUN"-gt 0]; thenecho already runningexit 1fi rsync-az vger.rutgers.edu:: cvs/CVSROOT/ChangeLog $ HOME/ChangeLog sum1 = 'sum $ HOME/ChangeLog 'sum2 = 'sum/var/www/cvs/vger/CVSROOT/ChangeLog 'if ["$ sum1 "=" $ sum2 "]; thenecho nothing to doexit 0fi rsync-az -- delete -- Force vger.rutgers.edu: cvs // var/www/cvs/vger/exit 06, FAQQ: how to perform rsync through ssh without entering a password? A: follow these steps to create an ssh keys on server A through SSH-keygen. do not specify A password ~ /. See identity and identity under ssh. pub file 2. create a subdirectory in the home directory on server B. ssh 3. set the identity of. copy pub to server B. 4. set identity. add pub ~ [User B]/. ssh/authorized_keys 5. as A result, user A on server A can use the following command to ssh user B to server B on the e.g. ssh-l userB serverB so that user A on server A can log on to server B as user B without A password. Q: How can I use rsync through the firewall without compromising security? A: The answer is as follows: either the server is in the firewall or the server is outside the firewall. In either case, ssh is usually used. In this case, it is best to create a backup user and configure sshd to only allow this user to access through RSA Authentication. If the server is in the firewall, it is best to limit the IP address of the client and reject all other connections. If the client is in the firewall, you can simply allow the firewall to open the ssh outbound connection on TCP port 22. Q: Can I back up the changed or deleted files? A: Of course. You can use rsync-other-options-backupdir =./backup-2000-2-13... This command is implemented. In this case, if the source file is/path/to/some/file. c changed, so the old file will be moved. /backup-2000-2-13/path/to/some/file. c. The directory needs to be manually created here. Q: which ports should I open on the firewall to adapt to rsync? A: Depending on the situation, rsync can directly transmit files through the tcp connection on port 873, or through ssh on port 22, but you can also use the following command to change its port: rsync-port 8730 otherhost: Or rsync-e 'ssh-p 2002 'otherhost: Q: how Can I copy only the directory structure through rsync and ignore the file? A: rsync-av-include '*/'-exclude '*' source-dir dest-dir Q: Why do I always see the "Read-only file system" error? A: Have you forgotten to set "read only = no"? Q: Why am I having the ERROR "@ ERROR: invalid gid? A: During rsync, uid = nobody is used by default; gid = nobody is used for running. If your system does not have A nobody group, this error will occur, you can try gid = nogroup or other Q: What is the failure to bind port 873? A: If you do not run the daemon with the root permission, this error will occur because the ports below port 1024 are privileged ports. You can use the-port parameter to change the value. Q: Why does my authentication fail? A: From the Perspective of your command line:> bash $ rsync-a 144.16.251.213: test> Password:> @ ERROR: auth failed on module test> I dont understand this. can somebody explain as to how to acomplish this.> all suggestions are welcome. there should be no issues caused by login with your username, try rsync-a max@144.16.251.213: test