Detailed description of file sync backup configuration using RSync

1. What is rsync?

Rsync, remote synchronize is a software that implements remote synchronization. It can keep the permissions, time, soft and hard links, and other additional information of the original file while Synchronizing files. Rsync uses the "rsync algorithm" to provide a fast way to synchronize files between clients and remote file servers. It can also transmit files through ssh, which ensures high confidentiality, in addition, it is free software.

Rsync includes the following features:

Update the entire directory and tree and file system;
Optional maintenance of symbolic chains, hard links, files belonging to, permissions, devices, and time;
No special permission requirements are required for installation;
For multiple files, the internal pipeline reduces the latency of file waiting;
You can use rsh, ssh, or direct port as the transfer port;
Supports anonymous rsync file synchronization and is an ideal image tool;

2. Set up an rsync server

Setting up an rsync server is relatively simple. Write a configuration file rsyncd. conf. There are also rules for writing files. We can refer to the documents on rsync.samba.org. Of course, we need to install the rsync software first;

A. Install rsync;

Obtain rsync

Rysnc Official Website: http://rsync.samba.org/you can get to the latest region from here. The latest version is 3.05. Of course, because rsync is such a useful software, many Linux releases include it.

Software Package Installation

# Sudo apt-get install rsync Note: Online installation methods such as debian and ubuntu;
# Yum install rsync Note: Online installation methods such as Fedora and Redhat;
# Rpm-ivh rsync Note: Install rpm packages such as Fedora and Redhat;

For other Linux distributions, use the corresponding software package management method.

　　Source code package installation

Tar xvf rsync-xxx.tar.gz
Cd rsync-xxx
./Configure -- prefix =/usr; make install Note: Before compiling and installing with the source code package, you must install gcc and compile and issue it;
　　　　
B. Configuration File

Rsync mainly includes the following three configuration files: rsyncd. conf (main configuration file), rsyncd. secrets (password file), rsyncd. motd (rysnc Server Information)

Server configuration file (/etc/rsyncd. conf). This file does not exist by default. Please create it.

　　The procedure is as follows:

# Touch/etc/rsyncd. conf # create rsyncd. conf, which is the configuration file of the rsync server.
# Touch/etc/rsyncd. secrets # create rsyncd. secrets, which is the user password file.
# Chmod 600/etc/rsyncd. secrets # Set the File Attribute of the rsyncd. secrets password file to root and set the permission to 600. Otherwise, the backup will fail!
# Touch/etc/rsyncd. motd

The next step is to modify the rsyncd. conf and rsyncd. secrets and rsyncd. motd files.

Set/etc/rsyncd. conf

Rsyncd. conf is the main configuration file of the rsync server. Let's take a simple example. The functions are described in detail later.

For example, if we want to back up/home And/opt on the server, I want to exclude the easylife and samba directories in/home;

# Distributed under the terms of the GNU General Public License v2
# Minimal configuration file for rsync daemon
# See rsync (1) and rsyncd. conf (5) man pages for help

# This line is required by the/etc/init. d/rsyncd script
Pid file =/var/run/rsyncd. pid
Port = 873
Address = 192.168.1.171
# Uid = nobody
# Gid = nobody
Uid = root
Gid = root

Use chroot = yes
Read only = yes

# Limit access to private LANs
Hosts allow = 192.168.1.0/255.255.255.0 10.0.1.0/255.255.255.0
Hosts deny = *

Max connections = 5
Motd file =/etc/rsyncd. motd

# This will give you a separate log file
# Log file =/var/log/rsync. log

# This will log every file transferred-up to 85,000 + per user, per sync
# Transfer logging = yes

Log format = % t % a % m % f % B
Syslog facility = local3
Timeout = 300

[Rhel4home]
Path =/home
List = yes
Ignore errors
Auth users = root
Secrets file =/etc/rsyncd. secrets
Comment = This is RHEL 4 data
Exclude = easylife/samba/

[Rhel4opt]
Path =/opt
List = no
Ignore errors
Comment = This is RHEL 4 opt
Auth users = easylife
Secrets file =/etc/rsyncd. secrets

Note: auth users is a real system user that must exist on the server. If you want to separate multiple users with commas (,), for example, auth users = easylife, root

Set Password File

The format of the password file is very simple. The content format of rsyncd. secrets is:

User name: Password

In this example, the content of rsyncd. secrets is similar to the following. In this document, some systems do not support long passwords. Please try setting them yourself.

Easylife: keer
Root: mike

Chown root. root rsyncd. secrets # modify owner
Chmod 600 rsyncd. secrets # Modify permissions

Note: 1. Set the File Attribute of the rsyncd. secrets password file to root and set the permission to 600. Otherwise, the backup will fail! For security purposes, the attributes of a file must be readable only by the owner.
2. The password here is worth noting. To ensure security, you cannot write the password of the System user here. For example, if your system user easylife password is 000000, you can set easylife in rsync to keer for security purposes. This is similar to the password principle for samba user authentication.

　Set the rsyncd. motd file;

It defines the information of the rysnc server, that is, the user logon information. For example, let the user know who provided the server. For example, when logging on to an ftp server, we see linuxsir.org ftp ....... Of course, this is not necessary for global definition of variables. You can use # To note or delete the variables. Here I wrote the content of rsyncd. motd:

++ ++
Welcome to use the mike.org.cn rsync services!
2002------2009
++ ++

Iii. Detailed configuration of rsyncd. conf Server

A. Global Definition

On the rsync server, there are several key global definitions, according to the previous configuration file rsyncd. conf;

Pid file =/var/run/rsyncd. pid Note: Tell the process to write to the/var/run/rsyncd. pid file;
Port = 873 Note: Specify the running port. The default value is 873. You can specify the port by yourself;
Address = 192.168.1.171 Note: Specify the Server IP address
Uid = nobody
Gid = nobdoy

Note: When the Server transfers files, the user and user group to be sent for execution is nobody by default. If you use nobody users and user groups, you may encounter permission issues. Some files cannot be pulled from the server. So I was lazy and used root for convenience. However, you can specify a user in the module defined when defining the directory to be synchronized to solve the permission problem.

Use chroot = yes

NOTE: With chroot, before transferring files, the server daemon sends chroot to the directory in the file system. This may protect the system from installation vulnerabilities. The disadvantage is that Super User Permissions are required. In addition, the symbolic link files will be excluded. That is to say, if you have signed a link on the rsync server, when you run the synchronization data of the client on the backup server, only the Symbolic Link name will be synchronized, the contents of the symbolic link are not synchronized. You need to try it yourself.

Read only = yes

Note: read-only is read-only, that is, the client is not allowed to upload files to the server. There is also a write only option. You can try it yourself;

# Limit access to private LANs
Hosts allow = 192.168.1.0/255.255.255.0 10.0.1.0/255.255.255.0

Note: You can specify a single IP address or the entire network segment to improve security. The format is separated by spaces between ip addresses and ip addresses, between ip addresses and CIDR blocks, and between CIDR blocks;

Max connections = 5

Note: Maximum number of client connections

Motd file =/etc/rsyncd. motd

Note: The motd file defines the server information. You must write the rsyncd. motd file Content yourself. This information is displayed when you log on. For example, I wrote:

++ ++
Welcome to use the mike.org.cn rsync services!
2002------2009
++ ++

Log file =/var/log/rsync. log

Note: rsync server logs;

Transfer logging = yes

Note: This is the log for transferring files.

Log format = % t % a % m % f % B
Syslog facility = local3
Timeout = 300

B. module definition

What does a module define? It mainly defines which directory on the server to be synchronized. Each module must be in the [name] format. This name is the name seen on the rsync client. It is actually a bit like the sharing name provided by the Samba server. The data actually synchronized by the server is specified through the path. You can specify multiple modules as needed. Each module must specify the authenticated user, password file, but the exclusion is not necessary

The following is an example of the preceding configuration file module:

[Rhel4home] # module it provides us with a link name, which is linked to the/home directory in this module; in the form of [name]

Path =/home # specifies the location of the file directory, which must be specified
Auth users = root # The authenticated user is root and must exist on the server
List = yes # list indicates whether the directory on the rsync server that provides synchronization data is displayed in the module on the server. The default value is yes. If you do not want to list the contents, no is required. If no is used, at least others do not know which directories are provided on your server. You just need to know it;
Ignore errors # ignore IO errors
Secrets file =/etc/rsyncd. secrets # Which file does the password have?
Comment = linuxsir home data # comments can be customized
Exclude = beinan/samba/

Note: exclude indicates exclusion. In other words, easylife and samba in the/home directory should be excluded. easylife/and samba/directories should be separated by spaces.

[Rhel4opt]
Path =/opt
List = no
Comment = optdir
Auth users = beinan
Secrets file =/etc/rsyncd. secrets
Ignore errors

4. Configure the rsync server and Firewall

It is quite simple to start the rsync server. There are several methods as follows:

A. -- daemon parameter mode: enables rsync to run in Server Mode

#/Usr/bin/rsync -- daemon -- config =/etc/rsyncd. conf # -- config is used to specify the location of rsyncd. conf. If it is in/etc, no write

B. xinetd Mode

Add the following content to modify services:
# Nano-w/etc/services

Rsync 873/tcp # rsync
Rsync 873/udp # rsync

This step can not be done. Generally, there are two rows (My RHEL4 and GENTOO both have by default ). The purpose of the modification is to let the system know the service name rsync corresponding to port 873. If not, add them on your own.

Set/etc/xinetd. d/rsync. A simple example is as follows:

# Default: off
# Description: The rsync server is a good addition to am ftp server, as it \
# Allows crc checksumming etc.
Service rsync
{
Disable = no
Socket_type = stream
Wait = no
User = root
Server =/usr/bin/rsync
Server_args = -- daemon
Log_on_failure + = USERID
}

Above, We need to enable the rsync daemon. Once an rsync client is connected, xinetd will forward it to rsyncd (port 873 ). Then the service xinetd restart takes effect.

Rsync server and Firewall

Linux Firewall uses iptables, so we should at least let the defined rsync server port pass through on the server side, and the client should also pass through.

# Iptables-a input-p tcp-m state -- state NEW-m tcp -- dport 873-j ACCEPT
# Iptables-L check if port 873 is enabled in the firewall

If you do not understand the firewall configuration, You can first disable the firewall by using service iptables stop. Of course, this is very dangerous in the production environment, so we can do this only when doing experiments!

5. Use the rsync client to synchronize data

A. Syntax explanation
　
After configuring the rsync server, you can issue the rsync command from the client to implement various synchronization operations. Rsync has many functions. The following describes common options:

The Command Format of rsync can be:
　　
1. rsync [OPTION]... SRC [SRC]... [USER @] HOST: DEST
2. rsync [OPTION]... [USER @] HOST: SRC DEST
3. rsync [OPTION]... SRC [SRC]... DEST
4. rsync [OPTION]... [USER @] HOST: SRC [DEST]
5. rsync [OPTION]... SRC [SRC]... [USER @] HOST: DEST
6. rsync [OPTION]... rsync: // [USER @] HOST [: PORT]/SRC [DEST]

Rsync has six different working modes:

1. Copy the local file. This mode is enabled when the SRC and DES paths do not contain a single colon ":" separator.
2. Use a remote shell program (such as rsh and ssh) to copy the content of the local machine to the remote machine. This mode is enabled when the DST path address contains a single colon ":" separator.
3. Use a remote shell program (such as rsh and ssh) to copy the contents of the remote machine to the local machine. This mode is enabled when the SRC address path contains a single colon ":" separator.
4. Copy files from the remote rsync server to the local machine. This mode is enabled when the SRC path information contains the ":" separator.
5. Copy files from the local machine to the remote rsync server. This mode is enabled when the DST path information contains the ":" separator.
6. List of remote machine files. This is similar to rsync transmission, but you only need to omit the local machine information in the command.
-A operates in archive mode, copies directories, and symbols to connect-rlptgoD

Parameters in rsync

-R is recursion
-L indicates a linked file, which means copying a linked file;-p indicates retaining the original permissions of the file;-t indicates retaining the original time of the file;-g indicates retaining the original user group of the file; -o: the original owner of the file;-D is equivalent to a block device file;
-Z compression during transmission;
-P transmission progress;
-V transmission progress and other information. It has something to do with-P. Try it on your own. You can see the document;
-E ssh parameters establish an encrypted connection.
-U is only updated to prevent new local files from being overwritten. Pay attention to the clock of both machines at the same time.
-- Progress indicates that the detailed progress is displayed.
-- Delete: if the server deletes this file, the client also deletes the file to ensure true consistency.
-- Password-file =/password/path/file to specify the password file, which can be used in the script without the need to enter the verification password interactively, note that only the owner can read the permission attribute of the password file.

B. Some Instances

B1. List the synchronization content provided on the rsync server;

First, let's take a look at the available data sources on the rsync server.

# Rsync -- list-only root@192.168.145.5 ::
++ ++
Welcome to use the mike.org.cn rsync services!
2002------2009
++ ++

Rhel4home This is RHEL 4 data

Note: The data source provided by rsync is the [rhel4home] module we wrote in rsyncd. conf. "This is RHEL 4 data" is provided by comment = This is RHEL 4 data in the [rhel4home] module. Why didn't we list the rhel4opt data source? Because we have already set list = no in [rhel4opt.

$ Rsync -- list-only root@192.168.145.5: rhel4home

++ ++
Welcome to use the mike.org.cn rsync services!
2002------2009
++ ++

Password:
Drwxr-xr-x 4096 2009/03/15 21:33:13.
-Rw-r -- 1018 2009/03/02 02:33:41 ks. cfg
-Rwxr-xr-x 21288 2009/03/15 21:33:13 wgetpaste
Drwxrwxr-x 4096 2008/10/28 21:04:05 cvsroot
Drwx ------ 4096 16:30:58 easylife
Drwsr-sr-x 4096 2008/09/20 22:18:05 giddir
Drwx ------ 4096 2008/09/29 14:18:46 quser1
Drwx ------ 4096 2008/09/27 14:38:12 quser2
Drwx ------ 4096 06:10:19 test
Drwx ------ 4096 2008/09/22 16:50:37 vbird1
Drwx ------ 4096 2008/09/19 15:28:45 vbird2

In the root @ ip address, root specifies the user name in the password file, followed by: rhel4home, which is the name of the rhel4home module.

B2, rsync client data synchronization;

# Rsync-avzP root@192.168.145.5: rhel4home rhel4home
Password: Enter the root Password provided by rsyncd. secrets on the server. In the previous example, we used mike. The entered password is not displayed. If you lose the password, press Enter.

Note: This command means to log on to the server as the root user and synchronize the rhel4home data to the local directory rhel4home. Of course, the local directory can be defined by yourself. If you do not have the rhel4home directory in the current operation directory on the client, the system will automatically create one for you. If the directory rhel4home exists, pay attention to its write permission.

# Rsync-avzP -- delete linuxsir@linuxsir.org: rhel4home rhel4home

This time, we introduce the -- delete option, indicating that the data on the client must be exactly the same as that on the server. If the linuxsirhome directory contains files that do not exist on the server, delete them. The ultimate goal is to make the data in the linuxsirhome directory completely consistent with that on the server; Be careful when using it; it is best not to treat the directory with important data as a local update directory, otherwise, all your data will be deleted;

Set rsync client

Set Password File

# Rsync-avzP -- delete -- password-file = rsyncd. secrets root@192.168.145.5: rhel4home rhel4home

This time we added the option -- password-file = rsyncd. secrets. This is when we log on to the rsync server as the root user to synchronize data, the password will read the rsyncd. secrets file. The content of this file is only the password of the root user. We need to do the following;

# Touch rsyncd. secrets
# Chmod 600 rsyncd. secrets
# Echo "mike"> rsyncd. secrets

# Rsync-avzP -- delete -- password-file = rsyncd. secrets root@192.168.145.5: rhel4home rhel4home

Note: The permission attribute of the password file must be set to only the primary readable.

In this way, no password is required. In fact, this is important because it is necessary for the server to schedule tasks through crond;

B3. Enable the rsync client to automatically synchronize data with the server

The server is a heavyweight application, so network backup of data is extremely important. We can configure the rsync server on the production server. We can use a machine with rysnc as a backup server. This Backup Server synchronizes data on the server at every day. Each backup is a complete backup. Sometimes the hard disk breaks down or the server data is deleted, and the complete backup is very important. This backup is equivalent to making an image for the server data every day. When an accident occurs on the production server, we can easily restore the data to minimize the data loss. Is that the case ??

Step 1: Create a synchronization script and Password File
　　
# Mkdir/etc/cron. daily. rsync
# Cd/etc/cron. daily. rsync
# Touch rhel4home. sh rhel4opt. sh
# Chmod 755/etc/cron. daily. rsync/*. sh
# Mkdir/etc/rsyncd/
# Touch/etc/rsyncd/rsyncrhel4root. secrets
# Touch/etc/rsyncd/rsyncrhel4easylife. secrets
# Chmod 600/etc/rsyncd/rsync .*

Note: Two files rhel4home. sh and rhel4opt. sh are created in/etc/cron. daily/and the permission is 755. Two password files are created. The root user uses rsyncrhel4root. secrets, And the easylife user uses rsyncrhel4easylife. secrets with the permission of 600;

We edit rhel4home. sh with the following content:

#! /Bin/sh
# Backup 192.168.145.5:/home
/Usr/bin/rsync-avzP -- password-file =/etc/rsyncd/rsyncrhel4root. password root@192.168.145.5: rhel4home/home/rhel4homebak/$ (date + '% m-% d-% y ')

Edit rhel4opt. sh with the following content:

#! /Bin/sh
# Backup 192.168.145.5:/opt
/Usr/bin/rsync-avzP -- password-file =/etc/rsyncd/rsyncrhel4easylife. secrets easylife@192.168.145.5: rhel4opt/home/rhel4hoptbak/$ (date + '% m-% d-% y ')

Note: You can merge the contents of rhel4home. sh and rhel4opt. sh into a file, for example, all of them are written to rhel4bak. sh;

Next we modify the content of/etc/rsyncd/rsyncrhel4root. secrets and rsyncrhel4easylife. secrets;

# Echo "mike">/etc/rsyncd/rsyncrhel4root. secrets
# Echo "keer">/etc/rsyncd/rsyncrhel4easylife. secrets

Then, create two directories rhel4homebak and rhel4optbak under the/home Directory, which means that the rhel4home data on the server is synchronized to/home/rhel4homebak on the backup server, rhel4opt data is synchronized to the/home/rhel4optbak/directory. Create directories based on years, months, and days; archive backups every day;

# Mkdir/home/rhel4homebak
# Mkdir/home/rhel4optbak

Step 2: modify the configuration file of the crond server and add it to the scheduled task.

# Crontab-e

Add the following content:

# Run daily cron jobs at 4: 10 every day backup rhel4 data:
10 4 ***/usr/bin/run-parts/etc/cron. daily. rsync 1>/dev/null

Note: The first line is a comment, indicating the content, so that you can remember it yourself.
The second line indicates that the executable script task under/etc/cron. daily. rsync is run at 04:10 every morning;
　　　　
After configuration, restart the crond server;

# Killall crond Note: process that kills the crond server;
# Ps aux | grep crond Note: Check whether it is killed;
#/Usr/sbin/crond Note: Start the crond server;
# Ps aux | grep crond Note: Check whether it is enabled?
Root 3815 0.0 0.0 1860 664? S/usr/sbin/crond
Root 3819 0.0 0.0 2188 808 pts/1 S + grep crond

Vi. FAQs

Q: How can I perform rsync through ssh without entering a password?

A: follow these steps:

1. Use ssh-keygen to create an SSH keys on server A. do not specify A password ~ /. Ssh: the identity and identity. pub files are displayed.
2. Create a subdirectory. ssh in the home directory on server B.
3. Copy identity. pub of a to server B.
4. Add identity. pub ~ [User B]/. ssh/authorized_keys
5. As A result, user A on server A can use the following command to ssh user B to server B. E.g. ssh-l userB serverB. In this way, user A on server A can log on to server B as user B without A password.

Q: How can I use rsync through the firewall without compromising security?
　　
A: The answer is as follows:

There are two common cases: one is that the server is inside the firewall and the other is outside the firewall. In either case, ssh is usually used. In this case, it is best to create a backup user and configure sshd to only allow this user to access through RSA Authentication. If the server is in the firewall, it is best to limit the IP address of the client and reject all other connections. If the client is in the firewall, you can simply allow the firewall to open the ssh outbound connection on TCP port 22.

Q: Can I back up the changed or deleted files?

A: Of course. You can use commands such as rsync-other-options-backupdir =./backup-2000-2-13. In this case, if the source file is/path/to/some/file. c changed, so the old file will be moved. /backup-2000-2-13/path/to/some/file. c. You need to manually create the directory here.

Q: What ports do I need to open on the firewall to adapt to rsync?
　
A: depends on the situation. Rsync can directly transfer files through tcp connection on port 873, or through ssh on port 22, but you can also change the port through the following command:
　　
Rsync -- port 8730 otherhost ::
Or
Rsync-e 'ssh-p 2002 'otherhost:

Q: How can I copy only the directory structure through rsync and ignore the file?
　　
A: rsync-av -- include '*/' -- exclude '* 'source-dir dest-dir

Q: Why do I always see the "Read-only file system" error?

A: Check if you forget to set "read only = no ".

Q: Why do I encounter the '@ ERROR: invalid gid' ERROR?

A: During rsync, uid = nobody is used by default; gid = nobody is used for running. If your system does not have A nobody group, this error will occur, you can try gid = ogroup or another

Q: Why does port 873 fail to be bound?
A: If you do not run the daemon with the root permission, this error will occur because the ports below port 1024 are privileged ports. You can use the -- port parameter to change the value.

Q: Why does my authentication fail?
A: From your command line: you are using

> Bash $ rsync-a 144.16.251.213: test
> Password:
> @ ERROR: auth failed on module test
>
> I dont understand this. Can somebody explain as to how to acomplish this.
> All suggestions are welcome.

There should be no issues caused by login with your username, try rsync-a max@144.16.251.213: test

Q: What is the following message?
@ ERROR: auth failed on module xxxxx
Rsync: connection unexpectedly closed (90 bytes read so far)
Rsync error: error in rsync protocol data stream (code 12) at I/O. c (150)

A: This is because the password is set incorrectly and cannot be successfully logged in. Please check the password settings in rsyncd. secrets again. Are the two sides consistent?

Q: What is the following message?

Password file must not be other-accessible
Continuing without password file
Password:

A: This indicates that the File Permission attribute of rsyncd. secrets is incorrect. It should be set to 600. Please download chmod 600 rsyncd. secrets

Q: What is the following message?

@ ERROR: chroot failed
Rsync: connection unexpectedly closed (75 bytes read so far)
Rsync error: error in rsync protocol data stream (code 12) at I/O. c (150)

A: This is usually because the directory set by path in your rsyncd. conf does not exist. Please use mkdir to open the Backup Directory first.

Complete!