Linux Remote Sync Tool rsync__linux

Rsync usually use a lot of places, find a detailed version, have various configuration instructions as well as several examples posted on the memo and reference

Rsync is a data mirroring Backup tool under Unix-like systems, and--remote sync can be seen from the name of the software. Its characteristics are as follows:

Can be mirrored to save the entire directory tree and file system.
It is easy to maintain the original file permissions, time, soft and hard links and so on.
can be installed without special permissions.
Optimized flow, high efficiency of file transfer.
You can use RCP, ssh, and so on to transfer files, of course, through a direct socket connection.
Support for anonymous transmission, to facilitate the use of Web site mirror image.
Software downloads
RYSNC's home address is:

http://rsync.samba.org/

The latest version is 3.0.8. You can choose to download from the original website: http://rsync.samba.org/ftp/rsync/.

This article is about 2.4.6

Compiling the installation
The compilation and installation of rsync is simple and requires only the following simple steps:

[Root@www rsync-2.4.6]#./configure
[Root@www rsync-2.4.6]# make
[Root@www rsync-2.4.6]# make install

However, it should be noted that rsync must be installed on servers A and B, where rsync is running in server mode on Server A and B on the client side of Rsync. This runs the rsync daemon on Web Server A and periodically runs the client on B to back up the content that needs to be backed up on Web server A.

Rsync Server
1, the start of the rsync server

On Web Server A, you need to start the rsync server in a daemon way, just run:

[Root@www rsync-2.4.6]#/usr/local/bin/rsync--daemon

Can be started. The Rsync default service port is 873, and the server receives a client's anonymous or authenticated backup request on that port.

There are several different ways to start a service at startup, such as:

A, join inetd.conf

Edit/etc/services, add rsync 873/tcp, specify the service port for Rsync is 873. Edit Add/etc/inetd.conf, add rsync stream TCP nowait root/bin/rsync rsync--daemon

Note: For xinetd, the setting method is similar.

B, Join rc.local

Edit/etc/rc.d/rc.local, add at the end:

/usr/local/bin/rsync--daemon

2, the configuration of rsync

For the rsync server, the most important and complex is its configuration. The rsync server's configuration file is/etc/rsyncd.conf, which controls authentication, access, logging, and so on.

The file is made up of one or more module structures. A module definition begins with the module name in square brackets until the next module definition begins or the file ends, and the module contains a parameter definition formatted with NAME = value. Each module actually corresponds to a directory tree that needs to be backed up, for example, in our instance environment, there are three trees that need to be backed up:/www/,/home/web_user1/and/home/web_user2/, then you need to define three modules in the configuration file. Corresponds to three trees respectively.

The configuration file is the behavior unit, which means that each new row represents a new annotation, module definition, or parameter assignment. A line that starts with a # represents a comment, and the line that ends with "" indicates that the following row is the continuation of the row. An equal sign in a parameter assignment may be a case-insensitive string, a Boolean value expressed in Trure/false.

Global parameters

All parameters before [Modlue] in a file are global parameters, and of course you can define module parameters in the global Parameters section, when the value of this parameter is the default value for all modules.

MOTD file

The MOTD file parameter is used to specify a message file that, when the client connects to the server, displays the contents of the file to the customer, and there is no MOTD file by default.

Log file

"Log File" specifies the log files for rsync and does not send the log to syslog.

PID File

Specifies the PID file for rsync.

Syslog facility

Specifies the message level for rsync when sending log messages to syslog, common message levels: Uth, Authpriv, cron, Daemon, FTP, Kern, LPR, mail, news, security, Sys-log, user, UUCP , Local0, Local1, Local2, Local3,local4, LOCAL5, Local6 and LOCAL7. The default value is daemon.

Module parameters

After a global parameter, you need to define one or more modules, and the following parameters can be defined in the module:

Comment

Assign a description to the module, which is displayed to the customer, along with the module name, when the client connects to the module list. The default does not describe the definition.

Path

Specifies the directory tree path for the module to be backed up, which must be specified.

Use Chroot

If "Use Chroot" is specified as true, rsync first chroot to the directory specified by the path parameter before transferring the file. The reason for this is to implement additional security protection, but the disadvantage is that you need to roots permissions, and you cannot back up the directory files that point to external symbolic connections. The chroot value is true by default.

Max connections

Specify the maximum number of concurrent connections to the module to protect the server, and a connection request exceeding the limit will be told to try again later. The default value is 0, which means there is no limit.

Lock file

Specifies the lock file that supports the Max connections parameter, and the default value is/var/run/rsyncd.lock.

Read Only

This option sets whether the customer is allowed to upload files. If true, any upload requests will fail, and if it is false and the server directory read-write permission is allowed, the upload is allowed. The default value is true.

List

This option sets whether the module should be listed when the client requests a list of modules that can be used. If you set this option to False, you can create a hidden module. The default value is true.

Uid

This option specifies the UID that the daemon should have when the module transmits files, with the GID option used to determine which file permissions to access, and the default value is "nobody".

Gid

This option specifies the GID that the daemon should have when the module transmits files. The default value is "nobody".

Exlude

Use to specify multiple pattern lists separated by spaces and add them to the exclude list. This is equivalent to using--exclude in the client command to specify the pattern, although the Exlude mode specified in the configuration file is not passed to the client and is applied only to the server. A module can specify only one exlude option, but it is possible to use "-" and "+" in front of the pattern to specify whether it is exclude or include.

However, it is important to note that this option has a certain security problem, the customer is likely to bypass the Exlude list, if you want to ensure that a particular file can not be accessed, it is best to combine the uid/gid option.

Exlude from

Specifies a filename that contains the definition of the Exclude mode from which the server reads the Exlude list definition.

Include

A list of patterns that specify multiple exlude that are separated by spaces and should be used. This equates to specifying patterns using--include in client commands, combining include and exlude to define complex exlude/include rules. A module can only specify an include option, but it is possible to use "-" and "+" in front of the pattern to specify whether it is exclude or include.

Include from

Specifies a file name that contains the definition of the include mode, from which the server reads the Include list definition.

Auth Users

This option specifies the list of user names separated by a space or comma, only those users who are allowed to connect to the module. There is no relationship between the user and the system user here. If "Auth users" is set up, then the client sends a connection request to the module and the challenged is requested by rsync to authenticate the Challenge/response authentication protocol used here. The user's name and password are stored in plaintext in the file specified by the Secrets file option. The module can be connected by default without a password (that is, anonymous).

Secrets file

This option specifies a file that contains a defined user name: a password pair. This file is only useful when "auth users" is defined. The file contains a username:passwd pair per line. Generally speaking, passwords are best not to exceed 8 characters. There is no default secures file name, you need to specify one in a limited style. (For example:/etc/rsyncd.secrets)

Strict modes

This option specifies whether to monitor the permissions of the password file, and if true, the password file can only be accessed by a user who is running the identity of the rsync server, and no other user can access the file. The default value is true.

Hosts allow

This option specifies which IP clients are allowed to connect to the module. The customer mode definition can be in the following form:

o xxx.xxx.xxx.xxx, the client host is allowed access only if it matches the IP exactly. For example: 192.167.0.1

O a.b.c.d/n, customers who belong to the network are allowed to connect to the module. For example: 192.168.0.0/24

O A.b.c.d/e.f.g.h, customers who belong to the network are allowed to connect to the module. For example: 192.168.0.0/255.255.255.0

o A host name that the client host is allowed to access only if it has that host name, for example: backup.linuxaid.com.cn.

o *.linuxaid.com.cn, all hosts that belong to this domain are allowed.

The default is to allow all host connections.

Hosts Deny

Specifies that machines that are not allowed to connect to the Rsync server can be defined by using the hosts allow definition. The default is no hosts deny definition.

Ignore errors

Specifies that RSYNCD ignores IP errors on the server when deciding whether to run a transfer when it is running, and in general Rsync will skip the--delete operation when an IO error occurs to prevent serious problems caused by temporary resource shortages or other IO errors.

Ignore nonreadable

Specifies that the RYSNC server completely ignores files that the user does not have access to. This is useful for situations where files that need to be backed up should not be available to a backup person.

Transfer logging

Enables RSYNC servers to use FTP-formatted files to record download and upload operations in their own separate logs.

Log format

This option allows users to customize the log file's fields using transfer logging. The format is a string that contains a format definition character, and the format definition you can use is as follows:

o%h Remote Host name

o%a Remote IP address

o%l file length character count

o%p Process ID for this rsync session

o%o Operation type: "Send" or "recv"

o%f filename

o%P Module Path

o%m Module Name

o%t Current Time

O%u authenticated username (null when anonymous)

o%b The actual number of bytes transferred

o%c when a file is sent, the field records the checksum code for that file

The default log format is: "%o%h [%a]%m (%u)%f%l", in general, "%t [%p]" is added to the header of each line. A Perl script called Rsyncstats is also published in the source code to count the log files in this format.

Timeout

This option allows you to override the IP timeout specified by the customer. This option ensures that the rsync server will not wait forever for a crashed customer. The timeout unit is seconds, and 0 indicates no time-out definition, which is also the default value. An ideal number for anonymous rsync servers is 600.

Refuse options

This option allows you to define a list of command arguments that are not allowed to be used by the customer for the module. You must use the full name of the command, not the abbreviation. However, when a command is rejected, the server reports an error message and exits. If you want to prevent the use of compression, it should be: "Dont compress = *".

Dont compress

Used to specify files that are not compressed for transfer, and the default value is

*.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz

Rsync Customer Command
After the rsync server configuration is over, the next step is to issue an rsync command on the client to enable the server-side files to be backed up to the client. Rsync is a very powerful tool, and its commands have many feature options, and we'll analyze all of its options below.

First, the command format for rsync can be:

rsync [OPTION] ... src [src] ... [user@] Host:dest

rsync [OPTION] ... [user@] HOST:SRC DEST

rsync [OPTION] ... src [src] ... DEST

rsync [OPTION] ... [user@] HOST::SRC [DEST]

rsync [OPTION] ... src [src] ... [user@] HOST::D EST

rsync [OPTION] ... rsync://[user@]host[:P ort]/src [DEST]
Rsync has six different modes of operation:

Copy local files and start this mode of work when both SRC and des path information do not contain a single colon ":" delimiter.

Use a remote shell program (such as rsh, SSH) to copy the contents of the local machine to the remote machine. This mode is started when the DST path address contains a single colon ":" Separator.

Use a remote shell program (such as rsh, SSH) to copy the contents of the remote machine to the local machine. This mode is started when the SRC address path contains a single colon ":" Separator.

Copy files to the local machine from a remote rsync server. This mode is started when the SRC path information contains the "::" delimiter.

Copy files from the local machine to the remote rsync server. This mode is started when the DST path information contains the "::" separator.

List of files for the remote machine. This is similar to rsync transmission, but simply omitting the local machine information in the command.
1. Usage

When transferring files using rsync, you need to specify a source and a purpose, one of which may be resource information for the remote machine. For example:

Rsync *.c foo:src/

Indicates that all the. C-terminated files under the current directory will be transferred to the SRC directory of machine foo. If any files already exist on the remote system, the remote update protocol is invoked to implement the transfer of only those files that have been updated.

Rsync-avz foo:src/bar/data/tmp

This command recursively transmits all content under the Src/bar directory on machine foo to the local/data/tmp/bar directory. Files are transmitted in an archive mode to ensure that symbolic links, attributes, permissions, owner, and so on are saved in transit. In addition, compression technology can be used to speed data transfer:

Rsync-avz foo:src/bar//data/tmp

The path information, at the end of "/", means copying the directory, not copying it at the end of "/". The difference between these two situations will be shown when the--delete option is used together.

You can also use rsync in local mode, if there is no ":" Symbol in the SRC and DST paths, which means that the command runs in local mode and is equivalent to the CP command.

Rsync somehost.mydomain.com::

This mode will list all the module information that somehost.mydomain.com can access.

Options description

-V,--verbose verbose mode output
-Q,--quiet thin output mode
-C,--checksum Open the check switch to force the file transfer to verify
-A,--archive archive mode, which represents the transfer of files recursively and maintains all file attributes equal to-rlptgod
-R,--recursive the subdirectory in recursive mode
-R,--relative using relative path information

Rsync foo/bar/foo.c remote:/tmp/

The foo.c file is created in the/tmp directory, and if the-r argument is used:

Rsync-r foo/bar/foo.c remote:/tmp/

The file/tmp/foo/bar/foo.c is created, which means that the full path information is persisted.

-B,--backup creates a backup, which means that the old file is renamed to ~filename for the same file name already exists for the purpose. You can use the--suffix option to specify different backup file prefixes.
--backup-dir store backup files (such as ~filename) in the directory.
-suffix=suffix defines the backup file prefix
-U,--update only updates, skipping all files that already exist in DST, and the file time is later than the file you want to back up. (Do not overwrite the updated file)
-L,--links retention soft link knot
-L,--copy-links to treat soft links like regular files
--copy-unsafe-links only copies links that point beyond the src Path directory tree
--safe-links ignores links to the SRC path tree
-H,--hard-links keep hard links
-P,--perms keep file permissions
-O,--owner keep file owner information
-G,--group keep file group information
-D,--devices maintain device file information
-T,--times keep file time information
-S,--sparse special handling of sparse files to save DST space
-N,--dry-run which files will be transmitted
-W,--whole-file copy files without incremental detection
-X,--one-file-system do not cross file system boundaries
-B, the block size used by the--block-size=size test algorithm is 700 bytes by default
-E,--rsh=command specifies a shell program that replaces rsh
--RSYNC-PATH=PATH Specifies the path information for the rsync command on the remote server
-C,--cvs-exclude use the same method as CVs to automatically ignore files to exclude files that you do not want to transfer
--existing updates only those files that already exist in DST, not the newly created files
--delete Delete files that are not in the DST SRC
--delete-excluded also deletes the receiving end of files that are excluded by the option specified
--delete-after after transmission is over
--ignore-errors in time IO errors are also deleted
--max-delete=num Delete NUM files up to
--partial retains files that are not fully transmitted for some reason, to expedite subsequent transmissions
--force forcibly deletes a directory, even if it is not empty
--numeric-ids does not match the user and group IDs of numbers to user and group names
--timeout=time IP timeout time in seconds
-I,--ignore-times not skip those files that have the same time and length
--size-only when deciding whether to back up a file, just look at the file size without considering the file time
--modify-window=num the time stamp window that determines whether a file is in the same time, default is 0
-t--temp-dir=dir create temporary files in dir
--compare-dest=dir also compares files in DIR to determine whether a backup is required
-P equals to--partial
--progress Display backup process
-Z,--compress compression of backed-up files on transfer
--EXCLUDE=PATTERN Specifies to exclude file modes that do not require transfer
--INCLUDE=PATTERN Specifies the file mode that is not excluded and needs to be transferred
--exclude-from=file exclude files in the specified mode in file
--include-from=file does not exclude files that match the file-specified pattern
--version Print version Information
--address bound to a specific address
--CONFIG=FILE specifies a different profile and does not use the default rsyncd.conf file
--PORT=PORT Specifies the other rsync service ports
--blocking-io blocking IO for a remote shell
-stats gives the transfer status of some files
--progress transmission process in real time
--log-format=format Specify log file format
--password-file=file gets the password from file
--bwlimit=kbps limit I/O bandwidth, Kbytes per second
-H,--help display Help information
Example analysis
This assumes that there are two servers: A and B. Where A is the main Web server, has domain name www.linuxaid.com.cn (202.99.11.120), b server is a backup machine, its domain name is backup.linuxaid.com.cn (202.99.11.121). The Web content of a is stored in several places:/www/and/home/web_user1/and/home/web_user2/. We need to establish a backup of the contents of these directories on backup machine B.

Server Configuration Instance

Then create the RSYNCD configuration file/etc/rsyncd.conf on the www.linuxaid.com.cn, as follows:

UID = nobody
gid = nobody use
chroot = no
max connections = 4
pid file =/var/run/rsyncd.pid
lock file =/var/run/rsync.lock
Log file =/var/log/rsyncd.log

[www]
path =/www/
Ignore errors
Read Only = TR UE
list = False
hosts allow = 202.99.11.121
hosts deny = 0.0.0.0/32
auth users = Backup
Secrets F ile =/etc/backserver.pas

[web_user1]
path =/home/web_user1/
Ignore errors
Read Only = True
List = False
hosts allow = 202.99.11.121
hosts deny = 0.0.0.0/32
uid = web_user1
gid = web_user1
Auth users = Backup
Secrets file =/etc/backserver.pas

[web_user2]
path =/home/web_user2/
-Ignore Errors
Read Only = True
list = False
hosts allow = 202.99.11.121
hosts deny = 0.0.0.0/32
uid = we B_user2
gid = web_user2
auth users = Backup
Secrets file =/etc/backserver.pas

There are four three modules defined here, respectively, corresponding to three directory trees that need to be backed up. This only allows 202.99.11.121 to back up native data and requires authentication. Three module-authorized backup users are backed up and the user information is stored in the file/etc/backserver.pas, which reads as follows:

backup:bk_passwd

And the file can only be read and written by the root user, otherwise the RSYNCD will fail when it starts. After these files are configured, you will need to start the RSYNCD server on Server A:

Rsync--daemon

Customer Command Example

/USR/LOCAL/BIN/RSYNC-VZRTOPG--delete--exclude "logs/"--exclude "conf/ssl.*/"--progress backup@202.99.11.120::www/ backup/www/--password-file=/etc/rsync.pass

The above command line in the-VZRTOPG V is verbose,z is compressed, R is RECURSIVE,TOPG is to maintain the original properties of the file, such as the owner, time parameters. --progress is the display of detailed progress,--delete means that if the server side to delete the file, then the client also deleted the file, to maintain true consistency. --exclude "logs/" indicates that the files under the/www/logs directory are not backed up. --exclude "conf/ssl.*/" indicates that the files under the/www/conf/ssl.*/directory are not backed up.

BACKUP@202.99.11.120::WWW indicates that the command is a backup of the WWW module in the server 202.99.11.120, and backup means that the module is backed up using Backup.

--password-file=/etc/rsync.pass to specify a password file so that you can use it in a script without having to enter the authentication password interactively, here's what you need to be aware of is that the password file permission attribute must be set to only root readable.

Here, the backup contents are stored in the/backup/www/directory of the backup machine.

[Root@linuxaid/]#/usr/local/bin/rsync-vzrtopg--delete--exclude "logs/"--exclude "conf/ssl.*/"--progress backup@202.99.11.120::www/backup/www/--password-file=/etc/rsync.pass
Receiving file list ... done
./
1
785 (100%)
1.py
4086 (100%)
2.py
10680 (100%)
A
0 (100%)
Ip
3956 (100%)
./
Wrote 2900 bytes read 145499 bytes 576.34 bytes/sec
Total size is 2374927 speedup is 45.34

The commands for operation of the other two modules are:

/USR/LOCAL/BIN/RSYNC-VZRTOPG--delete--progress backup@202.99.11.120::web_user1/backup/web_user1/--password-file =/etc/rsync.pass

/USR/LOCAL/BIN/RSYNC-VZRTOPG--delete--progress backup@202.99.11.120::web_user2/backup/web_user2/--password-file =/etc/rsync.pass

You can automate backups by using the CRONTAB-E command for customer commands, such as CRONTAB-E:

Some sample scripts
These scripts here are examples of rsync sites:

1, every seven days the data to the central server to do incremental backup

#!/bin/sh # This script does personal backups to a rsync backup server. You'll end up # with a 7 day rotating incremental backup. The incrementals'll go to subdirectories named after the day of the week, and the current # full backup goes into a  Directory called ' current ' # tridge@linuxcare.com directory to backup bdir=/home/$USER # excludes file-this contains A wildcard pattern/line of files to exclude excludes= $HOME/cron/excludes # The name of the backup machine Bserver=ow L # Your password on the backup server export rsync_password=xxxxxx ################################################### Backupdir= ' Date +%a ' opts= '--force--ignore-errors--delete-excluded--exclude-from= $EXCLUDES--#####################  -delete--backup--backup-dir=/$BACKUPDIR-a "export path= $PATH:/bin:/usr/bin:/usr/local/bin # The following line clears The last weeks incremental directory [-D $HOME/emptydir] | | mkdir $HOME/emptydir rsync--delete-a $HOME/emptydir/$BSERVER:: $USER/$BACKUpdir/rmdir $HOME/emptydir # Now the actual transfer rsync $OPTS $BDIR $BSERVER:: $USER/current 

2, Backup to a free hard drive

#!/bin/sh

export path=/usr/local/bin:/usr/bin:/bin

list= "rootfs usr data data2" for

D in $LIST;
mount/backup/$d
rsync-ax--exclude fstab--delete/$d//backup/$d/
umount/backup/$d
done

day= ' date ' +%a "'

rsync-a--delete/usr/local/apache/data2/backups/$DAY
rsync-a--delete/data/solid/data2/backups/$ Day

3, mirror the CVS Tree of vger.rutgers.edu

#!/bin/bash

cd/var/www/cvs/vger/
path=/usr/local/bin:/usr/freeware/bin:/usr/bin:/bin

RUN= ' LPs x | grep rsync | Grep-v grep | Wc-l '
if ["$RUN"-GT 0]; then
Echo already running
exit 1
fi

Rsync-az vger.rutgers.edu::cvs/c Vsroot/changelog $HOME/changelog

sum1= ' sum $HOME/changelog '
sum2= ' sum/var/www/cvs/vger/cvsroot/  Changelog '

If ["$sum 1" = "$sum 2"]; then
echo nothing todo
exit 0
fi

rsync-az--delete--force vger.rutgers.edu::cvs//var/www/cvs/vger/
Exit 0

FAQ
Q: How to use SSH for rsync without having to enter a password.
A: You can do this by following these steps

1. Set up SSH keys on Server A by Ssh-keygen, do not specify a password, you will see the identity and identity.pub files under ~/.ssh
2. Create subdirectories in the home directory on Server B. SSH
3. Copy A's identity.pub to Server B
4. Add Identity.pub to ~[user B]/.ssh/authorized_keys
5. A user on Server A can then SSH to Server B with User B via the following command
e.g. Ssh-l UserB ServerB
This allows user A on server A to SSH to User B's identity without a password to log on to Server B.

Q: How do I use rsync through a firewall without compromising security?
A: The answers are as follows:

This usually has two kinds of situations, one is the server inside the firewall, one is the server outside the fire wall. In either case, usually using SSH, it is best to create a new backup user and configure sshd only to allow this user to enter through the RSA authentication method. If the server is inside the firewall, it is best to qualify the client's IP address and reject all other connections. If the client is inside the firewall, it is OK to simply allow the firewall to open an SSH outgoing connection to TCP port 22.

Q: Can I back up any files that have been changed or deleted?
A: Of course you can:

You can use such as: Rsync-other-options-backupdir =./backup-2000-2-13 ... Such a command to implement.
So if the source file:/path/to/some/file.c changed, the old file will be moved to./BACKUP-2000-2-13/PATH/TO/SOME/FILE.C,
The catalogue here needs to be built by hand.

Q: What ports do I need to open on the firewall to accommodate rsync?
A: Depending on the situation

Rsync can transfer files directly through a TCP connection on port 873, or through 22-port SSH for file delivery, but you can also change its port via the following command:

Rsync--port 8730 otherhost::
Or
Rsync-e ' Ssh-p 2002 ' Otherhost:

Q: How can I just copy the directory structure through rsync and ignore the file?
A:rsync-av--include ' * * *--exclude ' * ' Source-dir dest-dir

Q: Why do I always have the "Read-only file system" error.
A: see if you forgot to set "Read Only = no"

Q: Why do I appear ' @ERROR: Invalid gid ' error.
A:rsync Use the default is to run with Uid=nobody;gid=nobody, if your system does not exist nobody group, there will be such a mistake, you can try the GID = Nogroup or other

Q: Bind Port 873 What's going on with failure.
A: If you are not running this daemon as root, this error can occur because the port below port 1024 is privileged. You can use the--port parameter to change.

Q: Why I failed to authenticate.
A: From your command line, it appears:

You use the following:
> bash$ rsync-a 144.16.251.213::test Test
> Password:
> @ERROR: Auth failed on module test
>
> I dont understand this. Can somebody explain as to the acomplish this.
> All suggestions are welcome.

There should be no problem with your username landing, try rsync-a max@144.16.251.213::test test