Rsync configuration details

Rsync configuration details this article describes 2.4.6 rsync compilation and installation is very simple, just need the following simple steps: [root @ www rsync-2.4.6] #. /configure [root @ www rsync-2.4.6] # make [root @ www rsync-2.4.6] # make install but note that you must install rsync on server A and server B, server A runs rsync in server mode, while server B runs rsync in client mode. In this way, the rsync daemon is run on web server A, and the client program is regularly run on B to back up the content to be backed up on web server. Rsync server 1. rsync server startup on web server A needs to start rsync server in daemon mode, just run: [root @ www rsync-2.4.6] #/usr/local/bin/rsync -- daemon to start. The default service port of rsync is 873. The server receives anonymous or authenticated backup requests from the customer on this port. There are several different methods to get the service up at startup, such as a and adding inetd. conf: edit/etc/services, add rsync 873/tcp, and set the service port of rsync to 873. Add/etc/inetd. conf and rsync stream tcp nowait root/bin/rsync -- daemon. Note: The setting method for xinetd is similar. B. Add rc. local edit/etc/rc. d/rc. local: Add/usr/local/bin/rsync -- daemon 2. rsync configuration is the most important and complex configuration for rsync servers. The configuration file of the rsync server is/etc/rsyncd. conf, which controls authentication, access, and logging. This file is composed of one or more modules. A module definition starts with the module name in square brackets until the definition of the next module starts or the file ends. The module contains the Parameter definition in the format of name = value. Each module corresponds to a directory tree to be backed up. For example, in our instance environment, there are three directory trees to be backed up: /www/,/home/web_user1/, And/home/web_user2/, You need to define three modules in the configuration file to correspond to three directory trees respectively. The configuration file is the unit of action, that is, each new line represents a new comment, module definition, or parameter value assignment. The row starting with # indicates the comment, and the row ending with "" indicates that the following row is the continuation of the row. After the parameter value is a medium number, it may be a case-insensitive string and a Boolean Value indicated by trure/false. In the file, all the parameters before [modlue] are global parameters. Of course, you can also define the module parameters in the global parameters section. At this time, the value of this parameter is the default value of all modules. The motd file "motd file" parameter is used to specify a message file. When the client connects to the server, the content of this file is displayed to the client. By default, there is no motd file. Log file "log file" specifies the log file of rsync, instead of sending the log to syslog. Pid file specifies the pid file of rsync. Syslog facility specifies the message level when rsync sends log messages to syslog. Common Message levels are: uth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, sys-log, user, uucp, local0, local1, local2, local3, local4, local5, local6, and local7. The default value is daemon. One or more modules need to be defined after global parameters. The module can define the following parameters: comment specifies a description for the module, this description is displayed to the customer when the customer connects to the module list. No description is defined by default. Path specifies the directory tree path for backup of this module. This parameter must be specified. Use chroot if "use chroot" is specified as true, rsync first chroot to the directory specified by the path parameter before transferring the file. The reason for doing so is to implement additional security protection, but the disadvantage is that you need to use the root privileges and cannot back up the directory files pointed to by the external symbolic connection. The default chroot value is true. Max connections specifies the maximum number of concurrent connections for this module to protect the server. connection requests that exceed the limit will be notified and then try again. The default value is 0, that is, there is no limit. Lock file specifies the lock file that supports the max connections parameter. The default value is/var/run/rsyncd. lock. Read only this option sets whether the client is allowed to upload files. If this parameter is set to true, all upload requests will fail. If this parameter is set to false and the server directory read/write permission permits upload. The default value is true. List this option sets whether the module should be listed when the customer requests a list of available modules. If this option is set to false, you can create a hidden module. The default value is true. Uid this option specifies the uid that the daemon should have when the module transfers files. With the gid option, you can determine which file permissions can be accessed. The default value is "nobody ". Gid this option specifies the gid that the daemon should have when the module transfers files. The default value is "nobody ". Exlude is used to specify multiple mode lists separated by spaces and add them to the exclude list. This is equivalent to using -- exclude to specify the mode in the Client Command. However, the exlude mode specified in the configuration file is not passed to the client, but only applied to the server. A module can only specify one exlude option, but you can use "-" and "+" before the mode to specify whether exclude or include. However, you must note that this option has certain security issues. The customer may bypass the exlude list. If you want to ensure that a specific file cannot be accessed, then it is best to use the uid/gid option together. Exlude from specifies a file name that contains the definition of the exclude mode. The server reads the definition of the exlude list from this file. Include is used to specify multiple rsyncs separated by spaces and the list of exlude modes. This is equivalent to using -- include in client commands to specify the mode. You can use include and exlude to define complex exlude/include rules. A module can only specify one include option, but you can use "-" and "+" before the mode to specify whether exclude or include. Include from specifies a file name that contains the definition of the include mode. The server reads the definition of the include list from this file. Auth users this option specifies a list of usernames separated by spaces or commas. Only these users can connect to this module. The user here has nothing to do with the System user. If "auth users" is set, the connection request sent by the client to this module will be sent by rsync to challenged for authentication. The challenge/response authentication protocol is used here. The user's name and password are stored in plaintext in the file specified by the "secrets file" option. By default, the module can be connected without a password (that is, the anonymous mode ). Secrets file this option specifies a file that contains a user name: Password pair. This file works only when "auth users" is defined. Each row of the file contains a username: passwd pair. Generally, the password should not exceed 8 characters. The default secures file name does not exist. You must specify a limit. (For example,/etc/rsyncd. secrets) strict modes this option specifies whether to monitor the permissions of the password file. If this option is set to true, the password file can only be accessed by users running identity on the rsync server, other users cannot access this file. The default value is true. Hosts allow this option specifies which IP addresses are allowed to connect to the module. The customer mode definition can be in the following format: o xxx. The customer host can only access this IP address if it exactly matches this IP address. For example, for 192.167.0.1 o a. B. c. d/n, all customers in this network can connect to this module. For example, 192.168.0.0/24 o a. B. c. d/e. f. g. h. Customers of this network can connect to this module. For example, if a host name is 192.168.0.0/255.255.255.0 o, the client host can only access the host name, for example, backup.linuxaid.com.cn. O * .linuxaid.com.cn, all hosts in this domain are allowed. By default, all hosts are allowed to connect. Hosts deny specifies a machine that is not allowed to connect to the rsync server. You can use hosts allow to define the host. Hosts deny is not defined by default. Ignore errors specifies that rsyncd ignores IP errors on the server when determining whether to delete a transmission operation. Generally, rsync skips the -- delete operation when an IO error occurs, to prevent serious problems caused by temporary lack of resources or other IO errors. Ignore nonreadable specifies that the rysnc server completely ignores files that users do not have access. This makes sense when some files in the directory to be backed up should not be backed up by the backup owner. Transfer logging enables rsync servers to use ftp files to record download and upload operations in their own separate logs. With this option, you can use transfer logging to customize log File fields. The format is a string containing the format specifiers. The format specifiers can be used as follows: o % h remote host name o % a remote IP address o % l file length characters o % p process id of the rsync Session o % o operation type: "send" or "recv" o % f file name o % P module path o % m Module name o % t current time o % u authenticated user name (null when anonymous) o % B actual number of bytes transferred o % c when a file is sent, this field records the file's verification code. The default log format is: "% o % h [% a] % m (% u) % f % l". Generally, "% t [% p]" is added to the header of each line. In the source code, a perl script program named rsyncstats is released to collect statistics on log files in this format. Timeout overwrites the specified IP timeout value. This option ensures that the rsync server will not always wait for a crashed customer. Timeout is measured in seconds. 0 indicates that no timeout is defined. This is also the default value. An ideal number for anonymous rsync servers is 600. Refuse options allows you to define a list of command parameters that cannot be used by customers for this module. The full name of the command must be used. However, when a command is rejected, the server reports an error message and then exits. To prevent compression, it should be: "dont compress = *". Dont compress is used to specify files that are not compressed and then transmitted. The default value is *. gz *. tgz *. zip *. z *. rpm *. deb *. iso *. bz2 *. after the customer command of tbz rsync finishes configuring the rsync server, the next step is to issue the rsync command on the client to back up the files on the server to the client. Rsync is a very powerful tool, and its commands have many special options. The following describes the options one by one. First, the rsync command format can be: rsync [OPTION]... SRC [SRC]... [USER @] HOST: DEST rsync [OPTION]... [USER @] HOST: src dest rsync [OPTION]... SRC [SRC]... DEST rsync [OPTION]... [USER @] HOST: SRC [DEST] rsync [OPTION]... SRC [SRC]... [USER @] HOST: DEST rsync [OPTION]... rsync: // [USER @] HOST [: PORT]/SRC [DEST] rsync has six different working modes: Copy local files; this mode is enabled when the SRC and DES paths do not contain a single colon ":" separator. A remote shell program (such as rsh and ssh) is used to copy the content of the local machine to the remote machine. This mode is enabled when the DST path address contains a single colon ":" separator. A remote shell program (such as rsh and ssh) is used to copy the contents of the remote machine to the local machine. This mode is enabled when the SRC address path contains a single colon ":" separator. Copy files from the remote rsync server to the local machine. This mode is enabled when the SRC path information contains the ":" separator. Copy files from the local machine to the remote rsync server. This mode is enabled when the DST path information contains the ":" separator. List of remote machine files. This is similar to rsync transmission, but you only need to omit the local machine information in the command. 1. When using rsync to transfer files, you must specify a source and a destination, one of which may be the resource information of the remote machine. For example, rsync *. c foo: src/transfers all files ending with. c in the current directory to the src directory of machine foo. If any file already exists in the remote system, the remote update protocol is called to transmit only the updated files. Rsync-avz foo: src/bar/data/tmp this command recursively transmits all contents in the src/bar directory on machine foo to the local/data/tmp/bar directory. Files are transmitted in archive mode to ensure that information such as symbolic links, attributes, permissions, and owner is saved during transmission. In addition, the compression technology can be used to speed up data transmission: rsync-avz foo: src/bar/data/tmp path information indicates copying the directory at the end, instead of ending with "/", the directory is copied. When the -- delete option is used together, the difference between the two cases is displayed. You can also use rsync in local mode. If the SRC and DST paths do not contain any ":" symbol, this command runs in local mode, which is equivalent to the cp command. Rsync somehost.mydomain.com: This mode lists all modules that can be accessed by somehost.mydomain.com. Option description-v, -- verbose detailed mode output-q, -- quiet streamlined output mode-c, -- checksum turn on the verification switch, force file transfer verification-, -- archive mode: Transfers files recursively and keeps all file attributes. equals to-rlptgoD-r, -- recursive processes subdirectories in recursive mode-R, -- relative uses the relative path information rsync foo/bar/foo. c remote:/tmp/create foo in the/tmp directory. c file, and if you use the-R parameter: rsync-R foo/bar/foo. c remote:/tmp/will create the file/tmp/foo/bar/foo. c, that is, the full path information will be maintained. -B, -- backup creates a backup, that is, if the object already has the same file name, rename the old file ~ Filename. You can use the -- suffix option to specify different backup file prefixes. -- Backup-dir: backs up files (for example ~ Filename) is stored in the directory. -Suffix = SUFFIX defines the backup file prefix-u. -- update only performs updates, that is, skipping all files that already exist in DST and whose file time is later than the backup time. (Do not overwrite the updated file)-l, -- links retains the soft link-L, -- copy-links: process soft links like regular files -- copy-unsafe-links: only copies links other than the SRC path directory tree -- safe-links ignores the directory tree pointing to the SRC path other links-H, -- hard-links keep hard links-p, -- perms keep File permissions-o, -- owner keep file owner information-g, -- group keep file group information-D, -- devices: Keep Device File Information-t, -- times: Keep file time information-S, -- sparse performs special processing on sparse files to save DST space-n, -- dry-run which files will be transmitted-W, -- whole-file to copy files without incremental detection-x, -- one-file-system: Do not span the file system boundary-B. -- block-size = SIZE indicates the block size used by the algorithm. The default value is Is 700 bytes-e, -- rsh = COMMAND specifies the shell program that replaces rsh -- rsync-path = PATH specifies the path of the rsync COMMAND on the remote server-C, -- cvs-exclude automatically ignores files in the same way as CVS to exclude files that do not want to be transmitted. -- existing only updates files that already exist in DST, instead of backing up those new files -- delete those files that are not in the dst src -- delete-excluded also delete the files that are excluded by this option at the receiving end -- after the delete-after transmission ends delete -- ignore-errors and delete it when I/o errors occur in a timely manner -- max-delete = NUM delete a maximum of NUM files -- partial keep those files that are not completely transmitted for any reason, to accelerate the subsequent re-transmission -- force Delete directory, even if it is not empty -- numeric-ids does not match the number user and group ID as the user Name and group name -- timeout = time ip timeout, in seconds-I, -- ignore-times does not skip files with the same time and length -- size-only. when determining whether to back up a file, only check the file size, regardless of the file time -- modify-window = NUM determines whether the file is time-stamped, the default value is 0-T -- temp-dir = DIR. Create a temporary file in DIR -- compare-dest = DIR and compare the files in DIR to determine whether to back up-P is equivalent to -- partial -- progress display. backup process-z, -- compress compresses the backup files during transmission -- exclude = PATTERN specifies to exclude the file mode that does not need to be transmitted -- include = PATTERN specifies the file mode to be transmitted without exclusion -- exclude- from = FILE exclude files in the specified mode -- include-from = FILE It is not excluded that the FILE specified to match the pattern -- version printing version information -- address is bound to a specific address -- config = FILE to specify other configuration files. The default rsyncd is not used. conf file -- port = PORT specify other rsync service ports -- blocking-io for remote shell use blocking IO-stats to give the transmission status of some files -- progress actual transmission process -- log -format = formAT specify the log file format -- password-FILE = FILE get the password from the file -- bwlimit = KBPS limit I/O bandwidth, KBytes per second-h, -- help displays help information for instance analysis. Here we assume there are two servers: A and B. A is the primary web server with the domain name www.linuxaid.com.cn (202.99.11.120), B is the backup machine, and its domain name is backup.linuxaid.com.cn (202.99.11.121 ). The web content of A is stored in/www/AND/home/web_user1/AND/home/web_user2 /. We need to back up the contents of these directories on backup machine B. Then, create the rsyncd configuration file/etc/rsyncd on www.linuxaid.com.cn. conf. Content: uid = nobodygid = nobodyuse chroot = nomax connections = 4pid file =/var/run/rsyncd. pidlock file =/var/run/rsync. locklog file =/var/log/rsyncd. log [www] path =/www/ignore errorsread only = truelist = falsehosts allow = 202.99.11.121hosts deny = 0.0.0.0/32 auth users = backupsecrets file =/etc/backserver. pas [web_user1] path =/ho Me/web_user1/ignore errorsread only = truelist = falsehosts allow = login deny = 0.0.0.0/32uid = web_user1gid = web_user1auth users = backupsecrets file =/etc/backserver. pas [web_user2] path =/home/web_user2/ignore errorsread only = truelist = falsehosts allow = login deny = 0.0.0.0/32uid = login = web_user2auth users = login file =/etc/backserver. pas set here There are four modules corresponding to the three directory trees to be backed up. Only the data of the local machine can be backed up at 202.99.11.121, and authentication is required. All backup users authorized by the three modules are backup, and user information is stored in the file/etc/backserver. in pas, the content is as follows: backup: bk_passwd and the file can only be read and written by the root user; otherwise, an error occurs when rsyncd is started. After these files are configured, you need to start the rsyncd server on server: rsync -- daemon Client Command example/usr/local/bin/rsync-vzrtopg -- delete -- exclude "logs/" -- exclude "conf/ssl. */"-- progress backup@202.99.11.120: www/backup/www/-- password-file =/etc/rsync. in the command line above, the v in vzrtopg is verbose, z is compressed, r is recursive, and topg is a parameter that maintains the original file attributes, such as the owner and time. -- Progress indicates that the detailed progress is displayed. -- delete indicates that if the server deletes the file, the client also deletes the file to ensure true consistency. -- Exclude "logs/" indicates that files in the/www/logs directory are not backed up. -- Exclude "conf/ssl. */" indicates that files in the/www/conf/ssl. */directory are not backed up. Backup@202.99.11.120: www indicates that the command is to back up the www module in the server 202.99.11.120, backup means to back up the module. -- Password-file =/etc/rsync. pass to specify the password file, so that the password can be used in the script without the need to enter the authentication password interactively. Note that the permission attribute of this password file must be set to only root readable. The backup content is stored in the/backup/www/directory of the backup machine. [Root @ linuxaid/] #/usr/local/bin/rsync-vzrtopg -- delete -- exclude "logs/" -- exclude "conf/ssl. */"-- progress backup@202.99.11.120: www/backup/www/-- password-file =/etc/rsync. passreceiving file list... done. /1785 (100%) 1. py4086 (100%) 2. py21380 (100%) a0 (100%) ip3956 (100% ). /wrote 2900 bytes read 145499 bytes 576.34 bytes/sectotal size is 2374927 speedup is 45.34 the commands for operating on the other two modules are:/usr/local/bin/r Sync-vzrtopg -- delete -- progress backup@202.99.11.120: web_user1/backup/web_user1/-- password-file =/etc/rsync. pass/usr/local/bin/rsync-vzrtopg -- delete -- progress backup@202.99.11.120: web_user2/backup/web_user2/-- password-file =/etc/rsync. pass can use the crontab-e command to implement automatic backup, for example, crontab-e: some sample scripts here are examples on the rsync Website: 1. Perform Incremental backup of data to the central server every seven days #! /Bin/sh # This script does personal backups to a rsync backup server. you will end up # with a 7 day rotating incremental backup. the incrementals will go # into subdirectories named after the day of the week, and the current # full backup goes into a directory called "current" # tridge@linuxcare.com # directory to backupBDIR =/home/$ USER # excludes file-this contains a wildcard pattern per line of fi Les to excludeEXCLUDES = $ HOME/cron/excludes # the name of the backup machineBSERVER = owl # your password on the backup serverexport RSYNC_PASSWORD = XXXXXX ########### ######################################## #################### BACKUPDIR = 'date + % a' OPTS = "-- force -- ignore-errors -- delete -excluded -- exclude-from = $ EXCLUDES -- delete -- backup-dir =/$ BACKUPDIR-a "export PATH = $ PATH: /bin:/usr /Local/bin # the following line clears the last weeks incremental directory [-d $ HOME/emptydir] | mkdir $ HOME/emptydirrsync -- delete-a $ HOME/emptydir/$ BSERVER:: $ USER/$ BACKUPDIR/rmdir $ HOME/emptydir # now the actual transferrsync $ OPTS $ BDIR $ BSERVER: $ USER/current 2. Back up to an idle hard disk #! /Bin/sh export PATH =/usr/local/bin:/usr/bin:/bin LIST = "rootfs usr data data2" for d in $ LIST; domount/backup/$ drsync-ax -- exclude fstab -- delete/$ d // backup/$ d/umount/backup/$ ddone DAY = 'date "+ % A" 'rsync -a -- delete/usr/local/apache/data2/backups/$ DAYrsync-a -- delete/data/solid/data2/backups/$ DAY 3. perform operations on the cvs tree of vger.rutgers.edu. image #! /Bin/bash cd/var/www/cvs/vger/PATH =/usr/local/bin:/usr/freeware/bin:/usr/bin: /bin RUN = 'lps x | grep rsync | grep-v grep | wc-l 'if ["$ RUN"-gt 0]; thenecho already runningexit 1fi rsync-az vger.rutgers.edu:: cvs/CVSROOT/ChangeLog $ HOME/ChangeLog sum1 = 'sum $ HOME/ChangeLog 'sum2 = 'sum/var/www/cvs/vger/CVSROOT/ChangeLog 'if ["$ sum1 "=" $ sum2 "]; thenecho nothing to doexit 0fi rsync-az -- delete -- Force vger.rutgers.edu: cvs // var/www/cvs/vger/exit 0 faq q: How to perform rsync over ssh without entering a password? A: follow these steps to create an ssh keys on server A through SSH-keygen. do not specify A password ~ /. See identity and identity under ssh. pub file 2. create a subdirectory in the home directory on server B. ssh3. set the identity of. copy pub to server B. 4. set identity. add pub ~ [User B]/. ssh/authorized_keys5. as A result, user A on server A can use the following command to ssh user B to server B on the e.g. ssh-l userB serverB so that user A on server A can log on to server B as user B without A password. Q: How can I use rsync through the firewall without compromising security? A: The answer is as follows: either the server is in the firewall or the server is outside the firewall. In either case, ssh is usually used. In this case, it is best to create a backup user and configure sshd to only allow this user to access through RSA Authentication. If the server is in the firewall, it is best to limit the IP address of the client and reject all other connections. If the client is in the firewall, you can simply allow the firewall to open the ssh outbound connection on TCP port 22. Q: Can I back up the changed or deleted files? A: Of course, you can use commands such as rsync-other-options-backupdir =./backup-2000-2-13. In this case, if the source file is/path/to/some/file. c changed, so the old file will be moved. /backup-2000-2-13/path/to/some/file. c. The directory needs to be manually created here. Q: which ports should I open on the firewall to adapt to rsync? A: Depending on the situation, rsync can directly transmit files through the tcp connection on port 873, or through ssh on port 22, but you can also use the following command to change its port: rsync -- port 8730 otherhost: Or rsync-e 'ssh-p 2002 'otherhost: Q: how Can I copy only the directory structure through rsync and ignore the file? A: rsync-av -- include '*/' -- exclude '* 'source-dir dest-dir Q: Why do I always see the "Read-only file system" error? A: Check if you forget to set "read only = no". Q: Why am I having an ERROR of '@ ERROR: invalid gid? A: During rsync, uid = nobody is used by default; gid = nobody is used for running. If your system does not have A nobody group, this error will occur, you can try gid = nogroup or other Q: What is the failure of binding port 873? A: If you do not run the daemon with the root permission, this error will occur because the ports below port 1024 are privileged ports. You can use the -- port parameter to change the value. Q: Why does my authentication fail? A: From the Perspective of your command line:> bash $ rsync-a 144.16.251.213: test> Password:> @ ERROR: auth failed on module test> I dont understand this. can somebody explain as to how to acomplish this.> all suggestions are welcome. there should be no issues caused by login with your username, try rsync-a max@144.16.251.213: test