rsync implementation of file synchronization backup configuration detailed



first, what is Rsync

Rsync,remote synchronize as implies to know it is a remote synchronization function of the software, it synchronized files at the same time, you can keep the original file permissions, time, soft and hard links and other additional information. Rsync provides a quick way to file synchronization between a client and a remote file server using the rsync algorithm, and can transmit files via SSH, which is also very confidential, and it is free software.

Rsync includes some of the following features:

Ability to update entire directories and trees and file systems;
Has the selective keeping symbol chain, the hard link, the document belongs, the authority, the equipment and the time and so on;
For installation, there is no special permission requirements;
For multiple files, the internal pipeline reduces the delay of file waiting;
Can be used rsh, SSH or direct port as the transmission into the port;
Support anonymous rsync synchronization files, is the ideal mirror tool;

second, the erection of rsync server

Set up Rsync server is simpler, write a configuration file rsyncd.conf. The writing of the file is also a rule, we can refer to the document on rsync.samba.org. Of course, we must first install rsync This software is only OK;

A, the installation of rsync;

Get rsync

RYSNC's official website: http://rsync.samba.org/can get the latest version from above. The current version is 3.05. Of course, because rsync is such a useful software, so many Linux distributions are included in it.

Package Installation

# sudo apt-get install rsync Note: Online installation methods in Debian, Ubuntu, etc.;
# yum Install rsync Note: Fedora, Redhat and other online installation methods;
# RPM-IVH rsync Note: Fedora, Redhat and other RPM package installation method;

Other Linux distributions, please install them using the appropriate package management method.

　　Source Pack Installation

Tar xvf rsync-xxx.tar.gz
CD rsync-xxx
./configure--prefix=/usr Make install Note: Before compiling the installation with the source package, you have to install the compilation of GCC and so on;
　　　　
B, configuration file

Rsync mainly has the following three profile rsyncd.conf (primary configuration file), rsyncd.secrets (password file), RSYNCD.MOTD (RYSNC server information)

Server configuration file (/etc/rsyncd.conf), the file does not exist by default, create it.

　　The specific steps are as follows:

#touch/etc/rsyncd.conf #创建rsyncd. conf, this is the configuration file for the rsync server.
#touch/etc/rsyncd.secrets #创建rsyncd. Secrets, this is the user password file.
#chmod 600/etc/rsyncd/rsyncd.secrets #将rsyncd. Secrets The file attribute of this password file is set to root, and the permissions are set to 600, otherwise it cannot be backed up successfully!
#touch/ETC/RSYNCD.MOTD

The next step is when we modify rsyncd.conf and Rsyncd.secrets and RSYNCD.MOTD files.

Set/etc/rsyncd.conf

Rsyncd.conf is the main configuration file for the rsync server. Let's start with a simple example, with a detailed description of each role.

For example, we want to back up the server/home and/opt, in the/home I want to exclude EasyLife and the Samba directory;

# Distributed under the terms of the GNU general public License v2
# Minimal configuration file for rsync daemon
# to Rsync (1) and rsyncd.conf (5) man pages for help

# This are required by THE/ETC/INIT.D/RSYNCD script
PID file =/var/run/rsyncd.pid
Port = 873
Address = 192.168.1.171
#uid = Nobody
#gid = Nobody
UID = root
GID = root

Use chroot = yes
Read Only = yes

#limit access to private LANs
Hosts allow=192.168.1.0/255.255.255.0 10.0.1.0/255.255.255.0
Hosts deny=*

Max connections = 5
MOTD file =/ETC/RSYNCD.MOTD

#This would give you a separate log file
#log file =/var/log/rsync.log

#This'll log every file transferred-up to 85,000+ per user, per sync
#transfer logging = yes

Log format =%t%a%m%f%b
Syslog facility = Local3
Timeout = 300

[Rhel4home]
Path =/home
List=yes
Ignore errors
Auth users = root
Secrets file =/etc/rsyncd.secrets
Comment = This is RHEL 4 data
Exclude = easylife/samba/

[Rhel4opt]
Path =/opt
List=no
Ignore errors
Comment = This is RHEL 4 opt
Auth users = EasyLife
Secrets file =/etc/rsyncd/rsyncd.secrets

Note: About auth users is the real system user that must exist on the server, if you want to use multiple users to separate the number, such as auth users = Easylife,root

Set Password file

The password file format is simple, and the Rsyncd.secrets content format is:

User name: Password

What we rsyncd.secrets in the example is similar to the following: In the document, some systems do not support long passwords and try to set them.

Easylife:keer
Root:mike

Chown Root.root rsyncd.secrets #修改属主
chmod rsyncd.secrets #修改权限

Note: 1, will rsyncd.secrets this password file file attribute to root owned, and permissions to set to 600, otherwise can not backup success! For security purposes, the properties of a file must be only owner-readable.
2, the password here is worth noting, in order to secure you can not write the password of the system user here. For example, your system user EasyLife password is 000000, for security you can let the EasyLife in Rsync as Keer. This is similar to the user-authenticated password principle of samba.

　set up RSYNCD.MOTD file;

It defines the RYSNC server information, which is the user login information. For example, let users know who the server is provided, and so on, similar to the FTP server login, we see the linuxsir.org ftp ... Of course, this is not necessary when defining variables globally, you can use the # number to note it, or delete it; I wrote a rsyncd.motd here that reads:

++++++++++++++++++++++++++++++++++++++++++++++
Welcome to use the mike.org.cn rsync services!
2002------2009
++++++++++++++++++++++++++++++++++++++++++++++

three, the rsyncd.conf server configuration detailed

A, global definition

In the rsync server, the global definition has several key, rsyncd.conf files according to the configuration file we gave earlier;

PID file =/var/run/rsyncd.pid Note: Tells the process to write to the/var/run/rsyncd.pid file;
Port = 873 Note: Specify the running ports, the default is 873, you can specify yourself;
Address = 192.168.1.171 Note: Specify server IP addresses
UID = nobody
GID = Nobdoy

Note: When the server-side transfer file, which user and user group to send to execute, default is nobody. If you use nobody users and user groups, you may experience permissions issues, and some files will not pull down from the server. So I was lazy, for convenience, with the root. However, you can resolve the issue of permissions by specifying users in the modules that are defined when you define the directory to synchronize.

Use chroot = yes

Note: With chroot, the server daemon will be chroot to the directory in the file system before the file is transferred, and the advantage of this is that it may protect the system from being installed with vulnerabilities. The disadvantage is that you need superuser privileges. In addition, the symbolic link file will be excluded. In other words, you are on the rsync server, if there is a symbolic link, you are running the client's synchronized data on the backup server, only the symbolic link name will be synchronized, and will not sync the contents of the symbolic link; this needs to be tried by itself.

Read Only = yes

Note: Read only is a read-only selection, that is, do not let the client upload files to the server. There is also a write only option, you try to do what use it;

#limit access to private LANs
Hosts allow=192.168.1.0/255.255.255.0 10.0.1.0/255.255.255.0

Note: When you can specify a single IP, you can also specify the entire network segment to improve security. The format is between IP and IP, IP and network segments, network segments and network segments between the space separated;

Max connections = 5

Note: Maximum number of client connections

MOTD file =/ETC/RSYNCD/RSYNCD.MOTD

Note: MOTD file is to define the server information, write the contents of RSYNCD.MOTD files yourself. This information is visible when the user logs on. For example, I wrote:

++++++++++++++++++++++++++++++++++++++++++++++
Welcome to use the mike.org.cn rsync services!
2002------2009
++++++++++++++++++++++++++++++++++++++++++++++

Log file =/var/log/rsync.log

Note: The log of rsync server;

Transfer logging = yes

Note: This is the log of the transfer file

Log format =%t%a%m%f%b
Syslog facility = Local3
Timeout = 300

B, module definition

What does the module define? The main is to define which directory the server is to be synchronized. Each module is in the form of [name]. The name is the name that is seen on the Rsync client, which is somewhat like the share name provided by the Samba server. The data that the server is actually synchronizing is specified through path. We can specify multiple modules according to our own needs. Each module to specify the authentication user, password file, but exclusion is not necessary

The following is an example of the previous configuration file module:

[Rhel4home] #模块它为我们提供了一个链接的名字, linked to the/home directory in this module;

Path =/home #指定文件目录所在位置, which must be specified
Auth users = Root #认证用户是root, is a user that must exist on the server
List=yes #list means that the directories that provide synchronized data on the Rsync server are listed on the server for the module. The default is yes. If you don't want to list it, no, if no is safe, at least someone else doesn't know what directories are available on your server. You know it.
Ignore Errors #忽略IO错误
Secrets file =/etc/rsyncd.secrets #密码存在哪个文件
Comment = Linuxsir home data #注释可以自己定义
Exclude = beinan/samba/

Note: Exclude is the meaning of exclusion, that is to say, to exclude easylife and samba in the/home directory; there are spaces between easylife/and samba/directories

[Rhel4opt]
Path =/opt
List=no
Comment = Optdir
Auth users = Beinan
Secrets file =/etc/rsyncd/rsyncd.secrets
Ignore errors

Four, start the rsync server and firewall settings

Starting the rsync server is fairly straightforward, and there are several ways to do this

A 、--daemon parameter mode, which is to allow rsync to run in server mode

#/usr/bin/rsync--daemon--config=/etc/rsyncd/rsyncd.conf #--config is used to specify the location of the rsyncd.conf, if the/etc can not write

B, Xinetd Way

Modify services to add the following content
# nano-w/etc/services

rsync 873/tcp # rsync
rsync 873/UDP # rsync

This step generally can not be done, usually have these two lines (my RHEL4 and Gentoo by default). The purpose of the modification is to let the system know that the 873 port corresponds to the service name Rsync. If not, join yourself.

Set the/etc/xinetd.d/rsync, the simple example is as follows:

# default:off
# description:the rsync server is a good addition to AM FTP server, as it \
#        allows CRC checksumming etc.
Service rsync
{
         Disable = no
        socket_type     = Stream
& nbsp;       wait             = no
        user             = root
        server           =/usr/bin/rsync
        server_args      =--daemon
        log_on_failure  + + USERID
}

Above, the main is to open rsync this daemon, once the rsync client to connect, XINETD will refer it to RSYNCD (port 873). Then the service xinetd restart to make the above setting effective.

rsync Servers and firewalls

Linux Firewall is used iptables, so we at least on the server side to let you define the Rsync server port through, the client should also let pass.

#iptables-A input-p tcp-m State--state new-m TCP--dport 873-j ACCEPT
#iptables-L To see if the firewall has 873 ports open

If you do not understand the configuration of the firewall, you can first service iptables stop to turn off the firewall. Of course in the production environment This is very dangerous, do experiments can do so yo.

v. Synchronizing data through the Rsync client

A, Grammar detailed
　
After the rsync server is configured, the Rsync commands can be issued from the client to implement various synchronization operations. Rsync has a number of feature options, and here's a list of common options:

The command format for rsync can be:
　　
1. rsync [OPTION] ... src [src] ... [user@] Host:dest
2. rsync [OPTION] ... [user@] HOST:SRC DEST
3. rsync [OPTION] ... src [src] ... DEST
4. rsync [OPTION] ... [user@] HOST::SRC [DEST]
5. rsync [OPTION] ... src [src] ... [user@] HOST::D EST
6. rsync [OPTION] ... rsync://[user@]host[:P ort]/src [DEST]

Rsync has six different modes of operation:

1. Copy local files and start this mode of work when both SRC and des path information do not contain a single colon ":" delimiter.
2. Use a remote shell program (such as rsh, SSH) to copy the contents of the local machine to the remote machine. This mode is started when the DST path address contains a single colon ":" Separator.
3. Use a remote shell program (such as rsh, SSH) to copy the contents of the remote machine to the local machine. This mode is started when the SRC address path contains a single colon ":" Separator.
4. Copy files from remote rsync server to local machine. This mode is started when the SRC path information contains the "::" delimiter.
5. Copy files from local machine to remote rsync server. This mode is started when the DST path information contains the "::" separator.
6. List of files for remote machines. This is similar to rsync transmission, but simply omitting the local machine information in the command.
-A operates in archive mode, replicates directories, symbolic connections equivalent to-rlptgod

Parameters in Rsync

-R is recursive
-L is a linked file, meaning copy linked file,-p to maintain the original permissions of the file,-T to maintain the original time of the file,-G to maintain the original user group,-O to maintain the original owner of the file,-D equivalent to block device files;
-Z compression when transmitting;
-P transmission progress;
-V The progress of the transmission and so on, and-p a little relationship, try it yourself. Can look at the document;
The-e SSH parameter establishes an encrypted connection.
-U only updates to prevent local new files from being rewritten, pay attention to both machine clocks while
--progress means showing detailed progress.
--delete means that if the server side deletes this file, then the client also deletes the file accordingly, maintaining true consistency
--password-file=/password/path/file to specify the password file so that you can use it in a script without having to enter the authentication password interactively, it is necessary to note that this password file permission attribute must be set to only the master readable.

B, some examples

B1, listing the provided synchronization content on the rsync server;

First: Let's look at what data sources are available on the rsync server

# rsync--list-only root@192.168.145.5::
++++++++++++++++++++++++++++++++++++++++++++++
Welcome to use the mike.org.cn rsync services!
2002------2009
++++++++++++++++++++++++++++++++++++++++++++++

Rhel4home This is RHEL 4 data

Note: Front is the data source provided by Rsync, which is the [rhel4home] module we wrote in Rsyncd.conf. The ' This is RHEL 4 data ' is provided by the comment = This is RHEL 4 data in the [Rhel4home] module; Why not list the rhel4opt data sources? Because we have list=no in [rhel4opt].

$ rsync--list-only Root@192.168.145.5::::rhel4home

++++++++++++++++++++++++++++++++++++++++++++++
Welcome to use the mike.org.cn rsync services!
2002------2009
++++++++++++++++++++++++++++++++++++++++++++++

Password:
Drwxr-xr-x 4096 2009/03/15 21:33:13.
-rw-r--r--1018 2009/03/02 02:33:41 ks.cfg
-rwxr-xr-x 21288 2009/03/15 21:33:13 wgetpaste
Drwxrwxr-x 4096 2008/10/28 21:04:05 cvsroot
drwx------4096 2008/11/30 16:30:58 EasyLife
Drwsr-sr-x 4096 2008/09/20 22:18:05 Giddir
drwx------4096 2008/09/29 14:18:46 Quser1
drwx------4096 2008/09/27 14:38:12 Quser2
drwx------4096 2008/11/14 06:10:19 test
drwx------4096 2008/09/22 16:50:37 vbird1
drwx------4096 2008/09/19 15:28:45 Vbird2

In the following root@ip, Root is the username in the specified password file, followed by:: Rhel4home This is the Rhel4home module name

B2, rsync client synchronization data;

#rsync-avzp root@192.168.145.5::rhel4home Rhel4home
Password: Enter the password for root, which is provided by the server-side rsyncd.secrets. In the previous example we used Mike, the input password is not back to show, the good to return.

Note: This command means to log on to the server with the root user and synchronize the Rhel4home data to the local current directory rhel4home. Of course, the local directory can be defined by yourself. If you do not rhel4home this directory in the directory under the current operation on the client, the system will automatically create one for you; when there is rhel4home this directory, you should pay attention to its write permission.

#rsync-avzp--delete linuxsir@linuxsir.org::rhel4home rhel4home

This time we introduce a--delete option to indicate that the data on the client is exactly the same as the server side, and if there are files in the Linuxsirhome directory that do not exist on the server, delete them. The ultimate goal is to make the data on the Linuxsirhome directory completely consistent with the server; be careful when you use it, it's best not to have an important number of directories as a local update directory, otherwise you will delete all your data;

Set up the Rsync client

Set Password file

#rsync-avzp--delete--password-file=rsyncd.secrets root@192.168.145.5::rhel4home rhel4home

This time we added an option--password-file=rsyncd.secrets, which reads rsyncd.secrets this file when we synchronize data with the root user logged on to the rsync server. The contents of this file are just the root user's password. We have to do the following;

# Touch Rsyncd.secrets
# chmod Rsyncd.secrets
# echo "Mike" > Rsyncd.secrets

# RSYNC-AVZP--delete--password-file=rsyncd.secrets root@192.168.145.5::rhel4home rhel4home

Note: Here it is necessary to note that this password file permission attribute must be set to only the owner-readable.

This does not require a password; in fact, this is more important, because the server through the Crond planning task is still necessary;

B3, allow rsync clients to automatically synchronize data with the server

The server is a heavyweight application, so the network backup of the data is extremely important. We can configure the Rsync server on the production server. We can take a RYSNC machine as a backup server. Have this backup server start synchronizing the data on the server every day at 4 o'clock in the morning, and each backup is a full backup. Sometimes the hard drive breaks down, or the server data is deleted, and a full backup is important. This backup is equivalent to a Daily mirror of the server's data, when the production server accident, we can easily recover the data, can reduce the loss of data to the minimum; that's not what happened.

Step1: Creating synchronization scripts and password files
　　
#mkdir/etc/cron.daily.rsync
#cd/etc/cron.daily.rsync
#touch rhel4home.sh rhel4opt.sh
#chmod 755/etc/cron.daily.rsync/*.sh
#mkdir/etc/rsyncd/
#touch/etc/rsyncd/rsyncrhel4root.secrets
#touch/etc/rsyncd/rsyncrhel4easylife.secrets
#chmod 600/etc/rsyncd/rsync.*

Note: We created two files rhel4home.sh and rhel4opt.sh in/etc/cron.daily/, and the permissions are 755. Created two password files root user is rsyncrhel4root.secrets, easylife user is rsyncrhel4easylife.secrets, permission is 600;

We edited the rhel4home.sh and the contents are as follows:

#!/bin/sh
#backup 192.168.145.5:/home
/usr/bin/rsync-avzp--password-file=/etc/rsyncd/rsyncrhel4root.password Root@192.168.145.5::rhel4home/home/rhel4 homebak/$ (date + '%m-%d-%y ')

We edited rhel4opt.sh, which reads:

#!/bin/sh
#backup 192.168.145.5:/opt
/usr/bin/rsync-avzp--password-file=/etc/rsyncd/rsyncrhel4easylife.secrets Easylife@192.168.145.5::rhel4opt/home /rhel4hoptbak/$ (date + '%m-%d-%y ')

Note: You can combine the contents of rhel4home.sh and rhel4opt.sh into a single file, such as all written in rhel4bak.sh;

Then we revise the contents of/etc/rsyncd/rsyncrhel4root.secrets and rsyncrhel4easylife.secrets;

# echo "Mike" >/etc/rsyncd/rsyncrhel4root.secrets
# echo "Keer" >/etc/rsyncd/rsyncrhel4easylife.secrets

Then we create the Rhel4homebak and Rhel4optbak two directories under the/home directory, meaning that the server-side rhel4home data is synchronized to the/home/rhel4homebak on the backup server, rhel4opt data is synchronized to/ The home/rhel4optbak/directory. and file the creation of the directory by month and day, daily backup is archived;

#mkdir/home/rhel4homebak
#mkdir/home/rhel4optbak

Step2: Modify the Crond server's configuration file to add to the scheduled task

#crontab-E

Add the following content:

# Run Daily cron jobs in 4:10 every day backup Rhel4 data:
4 * * */usr/bin/run-parts/etc/cron.daily.rsync 1>/dev/null

Note: The first line is the annotation, is the description content, so can remember.
The second line indicates that the executable Script task under/etc/cron.daily.rsync is run at 4:10 A.M. every day;
　　　　
After the configuration, to restart the Crond server;

# killall Crond Note: The process of killing the Crond server;
# PS aux |grep crond Note: Check to see if killed;
#/usr/sbin/crond Note: start the Crond server;
# PS aux |grep crond Note: Check to see if it's started.
Root 3815 0.0 0.0 1860 664? S 14:44 0:00/usr/sbin/crond
Root 3819 0.0 0.0 2188 808 pts/1 s+ 14:45 0:00 grep crond

Vi. FAQ

Q: How to use SSH for rsync without having to enter a password.

A: You can do this by following these steps

1. Set up SSH keys on Server A by Ssh-keygen, do not specify a password, you will see the identity and identity.pub files under ~/.ssh
2. Create subdirectories in the home directory on Server B. SSH
3. Copy A's identity.pub to Server B
4. Add Identity.pub to ~[user B]/.ssh/authorized_keys
5. A user on Server A can then SSH to Server B with User B via the following command. e.g. Ssh-l UserB ServerB. This allows user A on server A to SSH to User B's identity without a password to log on to Server B.

Q: How do I use rsync through a firewall without compromising security?
　　
A: The answers are as follows:

This usually has two kinds of situations, one is the server inside the firewall, one is the server outside the fire wall. In either case, usually using SSH, it is best to create a new backup user and configure sshd only to allow this user to enter through the RSA authentication method. If the server is inside the firewall, it is best to qualify the client's IP address and reject all other connections. If the client is inside the firewall, it is OK to simply allow the firewall to open an SSH outgoing connection to TCP port 22.

Q: Can I back up any files that have been changed or deleted?

A: Of course you can. You can use such as: Rsync-other-options-backupdir =./backup-2000-2-13 ... Such a command to implement. So if the source file:/path/to/some/file.c changed, the old file will be moved to./backup-2000-2-13/path/to/some/file.c, this directory needs to be built by hand.

Q: What ports do I need to open on the firewall to accommodate rsync?
　
A: Depending on the situation. Rsync can transfer files directly through a TCP connection on port 873, or through 22-port SSH for file delivery, but you can also change its port via the following command:
　　
Rsync--port 8730 otherhost::
Or
Rsync-e ' Ssh-p 2002 ' Otherhost:

Q: How can I just copy the directory structure through rsync and ignore the file?
　　
A:rsync-av--include ' * * *--exclude ' * ' Source-dir dest-dir

Q: Why do I always have the "Read-only file system" error.

A: see if you forgot to set "Read Only = no"

Q: Why do I appear ' @ERROR: Invalid gid ' error.

A:rsync Use the default is to run with Uid=nobody;gid=nobody, if your system does not exist nobody group, there will be such a mistake, you can try the GID = ogroup or other

Q: Bind Port 873 What's going on with failure.
A: If you are not running this daemon as root, this error can occur because the port below port 1024 is privileged. You can use the--port parameter to change.

Q: Why I failed to authenticate.
A: From your command line, you are using

> bash$ rsync-a 144.16.251.213::test Test
> Password:
> @ERROR: Auth failed on module test
>
> I dont understand this. Can somebody explain as to the acomplish this.
> All suggestions are welcome.

There should be no problem with your username landing, try rsync-a max@144.16.251.213::test test

Q: What happened to the following message?
@ERROR: Auth failed on module xxxxx
Rsync:connection unexpectedly closed (bytes read so far)
Rsync Error:error in Rsync protocol The data Stream (code) at IO.C (150)

A: This is because the password is set incorrectly, can not login successfully, please check the password set in the Rsyncd.secrets, the two end is consistent?

Q: What happened to the following message?

Password file must not is other-accessible
Continuing without password file
Password:

A: This means that the Rsyncd.secrets file permissions attribute is incorrect and should be set to 600. Please lower chmod rsyncd.secrets

Q: What happened to the following message?

@ERROR: Chroot failed
Rsync:connection unexpectedly closed (bytes read so far)
Rsync Error:error in Rsync protocol The data Stream (code) at IO.C (150)

A: This is usually not the same directory that is set up in your path path in your rsyncd.conf. Please open the backup directory with mkdir first.