Rotten mud: rsync learning for linux file synchronization (I), linuxrsync

Rotten mud: rsync learning for linux file synchronization (I), linuxrsync

This article was sponsored by Xiuyi linfeng and first launched in the dark world.

When we are free these days, we plan to start learning the file sync software rsync in linux. When learning rsync, we can perform the following steps:

1. What is rsync?

2. How rsync works

3. Advantages of rsync

4. rsync Authentication

5. Install rsync

6. Explanation of rsync Command Parameters

7. Configure rsync

8. rsync startup and Shutdown

9. rsync synchronization instance

Note: The OS used by the rsync server and client is centos 6.5 64bit.

I. What is rsync?

Before getting started with rsync, let's answer this question: what is rsync.

Rsync (remote synchronize) is a remote data synchronization tool in Liunx/Unix. It can quickly synchronize files and directories between multiple hosts through the LAN/WAN, and use the rsync algorithm (differential encoding) to reduce data transmission.

The rsync algorithm does not transmit the entire file every time, but only transmits different parts of the two files, so the transmission speed is quite fast.

In addition, rsync can copy, display directory attributes, copy files, and selectively compress and recursively copy files.

Ii. How rsync works

1. The client constructs a FileList. FileList contains the name-> id (id is used to uniquely represent a file, such as MD5), for all file information to be synchronized with the server ).

2. The client sends FileList to the server.

3. on the server, rsync processes the FileList sent from the client and creates a new NewFileList. Based on the MD5 Value Comparison, Delete the existing file information pairs on the server, and only keep the files that do not exist or change on the server.

4. The client obtains the NewFileList sent by the server, and then transfers the file in NewFileList to the server.

Iii. Advantages of rsync

Rsync has the following advantages:

1) images can be used to save the entire directory tree and file system.

2) It is easy to maintain the permissions, time, and soft-hardware connections of the original file.

3) installation without special permissions.

4) optimized processes and high file transfer efficiency.

5) You can use shell (rsh, ssh) to transmit files.

6) Anonymous operation is supported.

7) compared with scp, the rsync transmission speed is definitely far higher than the scp transmission speed.

We often use rsync and scp in the LAN to transmit a large number of mysql database files, and found that rsync transfers files at least 20 times faster than scp.

Therefore, if you need to transfer massive data between Liunx and Unix servers, you are advised to select rsync for transmission.

Iv. rsync Authentication

Rsync has two common authentication methods: rsync-daemon and ssh. During normal use, we use the rsync-daemon method most.

Note: When rsync is used, the rsync program must be installed on both the server and client..

4.1 rsync-daemonAuthentication

Rsync listens to TCP port 873 by default in rsync-daemon Authentication mode.

Rsync-daemon authentication is the main authentication method of rsync, which is also a frequently used authentication method. In this mode, rsync can write the password to a file.

Note: For rsync-daemon authentication, you must install the rsync service on both the server and client. You only need to start rsync on the rsync server and configure the rsync configuration file. The client does not start the rsync service and does not affect normal synchronization.

Ssh 4.2Authentication

In the ssh Authentication mode, rsync can be authenticated by system users, that is, transmitted through the ssh tunnel on rsync, similar to the scp tool. The synchronization operation is not limited to the synchronization folder defined in rsync.

Note: For ssh authentication, you do not need to configure rsync on the server or client.Configuration File, You only need to install the rsync service on both sides, and do not need to start rsync on both sides.

If SSH on the rsync server is a standard port, rsync is used as follows:

Rsync-avz/root/test root@192.168.199.248:/root/

If the SSH port on the rsync server is a non-standard port, you can use the-e parameter of rsync to specify the port. The usage is as follows:

Rsync-avz/root/test-e 'ssh-p1234' root@192.168.199.248:/root/

5. Install rsync

To install rsync, you can install rsync in two ways: source code installation and RPM installation.

Note: The rsync software is the same software package on both the server side and the client side.

The following describes the two installation methods.

5.1Source code Installation

To install rsync using the source code, we need to download the corresponding installation package from its official website. Official rsync Website: rsync.samba.org. As follows:

We use wget for download, as shown below:

Wget http://rsync.samba.org/ftp/rsync/rsync-3.1.1.tar.gz

After the installation package is downloaded, decompress and install it. As follows:

Tar-xf rsync-3.1.1.tar.gz

./Configure

Make & make install

Note: When installing rsync in the source code, the gcc library files required for compiling should be installed as early as possible.

The rsync source code installation will install rsync in the/usr/local/directory by default. We can see it through the help. As follows:

./Configure -- help

After rsync is installed, you can view rsync help. Switch to the/usr/local/bin directory as follows:

/Usr/local/bin

Rsync-h

If you want to enable rsync installed in source code. We can directly put the startup command into the rc. local file. As follows:

Echo "/usr/local/bin/rsync -- daemon -- config =/etc/rsyncd. conf">/etc/rc. local

You can also write it as follows:

Echo "/usr/local/bin/rsync -- daemon">/etc/rc. local

Cat/etc/rc. local

5.2 RPMInstallation Method

Rsync installation in RPM mode is relatively simple. You can directly install rsync using yum. As follows:

Yum-y install rsync

Check the rsync installation location as follows:

Rpm-SQL rsync-3.0.6-12.el6.x86_64

You can see that rsync installed in RPM mode installs rsync in the/usr/bin directory, and generates a/etc/xinetd. d/rsync file.

View the help documentation generated during installation as follows:

More/usr/share/doc/rsync-3.0.6/README

After the installation is complete, you can view the directory where the rsync command is located and the help of rsync. As follows:

Which rsync

Rsync -- help

It is obvious that rsync is indeed installed in the/usr/bin directory.

If you want to enable rsync installed in RPM to start up, you can also enable it to start up like the source code.

In addition, there is also a way to enable it to boot. If you observe it, you will find the/etc/xinetd. d/rsync file generated during rsync installation.

In centos, rsync runs the rsync service in xinetd mode by default. If you install rsync in RPM mode, you only need to start the xinet service.

However, before starting the xinetd service, we also need the configuration file/etc/xinetd. d/rsync, as shown below:

Vi/etc/xinetd. d/rsync

After the configuration is complete, we also need to install the xinetd package. Otherwise, the xinetd service cannot be started. As follows:

Yum-y install xinetd

/Etc/init. d/xinetd start

Chkconfig xinetd on

Netstat-tunlp | grep 873

6. Explanation of rsync Command Parameters

After rsync is installed, let's check the help of the rsync Command, as shown below:

Rsync -- help

6.1 rsyncSeveral important parameters

Here we will give a general introduction to several frequently used parameters:-v,-a, and-z. For detailed descriptions of these parameters, see the article "rotten mud: [Switch] detailed explanation of rsync command parameters.

-V, -- verbose detailed mode output.

-A, -- archive mode, indicating that the file is transmitted recursively and the attributes of all files remain unchanged.

-Z, -- compress compresses the backup files during transmission.

-- Delete: delete files that exist in DST but do not exist in SRC.

6.2 rsyncSix working modes

In addition to the above parameters, we also noticed the seven command formats of rsync:

1) rsync [OPTION]... SRC [SRC]... DEST

2) rsync [OPTION]... SRC [SRC]... [USER @] HOST: DEST

3) rsync [OPTION]... SRC [SRC]... [USER @] HOST: DEST

4) rsync [OPTION]... SRC [SRC]... rsync: // [USER @] HOST [: PORT]/DEST

5) rsync [OPTION]... [USER @] HOST: SRC [DEST]

6) rsync [OPTION]... [USER @] HOST: SRC [DEST]

7) rsync [OPTION]... rsync: // [USER @] HOST [: PORT]/SRC [DEST]

The seven command formats represent six different working modes of rsync, with the fourth and seventh modes not significantly different. Among these modes, the third and sixth modes are frequently used, especially the third mode.

The six modes can be divided by two words: push and pull.

Push is to execute the rsync command on the client to push the files to be synchronized from the client to the server.

The rsync command is also executed on the client to pull files from the server to the local machine.

Note: The rsync command is executed on the client, whether it is push or pull, but the command format is different.

1) rsync [OPTION]... SRC [SRC]... DEST

Synchronize local files from one directory to another. For example, rsync-avz/data/backup means to synchronize files in the local/data directory to the local/backup directory.

2) rsync [OPTION]... SRC [SRC]... [USER @] HOST: DEST

A remote shell program (such as rsh and ssh) is used to synchronize local files to a remote machine.This method is a push method.For example: rsync-avz/data test@192.168.199.247:/backup, indicates to synchronize files in the local/data directory to the/backup Directory of the server 192.168.199.247.

3) rsync [OPTION]... SRC [SRC]... [USER @] HOST: DEST

Synchronize local files to remote serversDEST indicates the Authentication Module name of the rsync server. This method is a push method.For example: rsync-avz/data test@192.168.199.247: backup -- password-file =/etc/rsyncd. password, which means to synchronize files in the local/data directory to the path under the backup module of the server 192.168.199.247.

4) rsync [OPTION]... SRC [SRC]... rsync: // [USER @] HOST [: PORT]/DEST

List objects on a remote machine. This is similar to the ls command, but you only need to omit the local machine information in the command. For example: rsync-v rsync: // test@192.168.199.247/backup, indicates that the contents of the/backup Directory of the server 192.168.199.247 are listed on the local machine, as follows:

5) rsync [OPTION]... [USER @] HOST: SRC [DEST]

Synchronize the files on the remote machine to the local machine,This method is a pull method.. For example: rsync-avz test@192.168.199.247:/backup/data, indicates to synchronize the files under the/backup Directory of 192.168.199.247 to the local/data directory.

6) rsync [OPTION]... [USER @] HOST: SRC [DEST]

Synchronize the files on the remote machine to the local machine,This method is a pull method.. For example: rsync-avz test@192.168.199.247: backup -- password-file =/etc/rsyncd. password/data: Synchronize the files in the path of the backup module of 192.168.199.247 to the local/data directory.

7. Configure rsync

After rsync is installed, We need to configure rsync. The most important and complex rsync server is its configuration.

The configuration of rsync is divided into the server side and the client side. Next we will explain it separately.

7.1 rsyncServer Configuration

The rsync server requires two configuration files: rsyncd. conf and rsyncd. password.

Rsyncd. conf is stored in the/etc/directory by default. It is also the main configuration file of the rsync server. This file is configured with the rsync server's control authentication, access, logging, and so on. Rsyncd. password is used to store the rsync user name and password.

However, after rsync is installed, the above two configuration files will not be generated. We need to create them manually.

The content of the rsyncd. conf configuration file is as follows:

Cat/etc/rsyncd. conf

Uid = root

Gid = root

User chroot = no

Max connections = 200

Timeout = 600

Pid file =/var/run/rsyncd. pid

Lock file =/var/run/rsyncd. lock

Log file =/var/run/rsyncd. log

[Backup]

Path =/backup/

Ignore errors

Read only = no

List = no

Hosts allow = 192.168.199.0/255.255.255.0

Auth users = test

Secrets file =/etc/rsyncd. password

[Www]

Path =/www/

Ignore errors

Read only = no

List = no

Hosts allow = 192.168.199.0/255.255.255.0

Auth users = apache

Secrets file =/etc/rsyncd. password

For details about the rsync configuration file, see the next article "mud: rsync configuration file details".

RsyncBy default, it runs under the nobody user, but we will let it run under the root user so that we don't worry about various permissions in the future.

After the configuration file is created, create the password file. As follows:

Echo "test: test">/etc/rsyncd. password

Echo "apache: apache">/etc/rsyncd. password

Cat/etc/rsyncd. password

Note: The user and password in the password file can be customized, and the user does not have to exist on the server.

After the password file is created, you must set its permissions to 600. Otherwise, a verification error will be prompted during future synchronization. As follows:

Chmod 600/etc/rsyncd. password

An rsync configuration file can contain multiple authentication modules, and a password file can also store multiple users and their corresponding passwords. Each authentication module can correspond to different clients.

After the preceding two files are created, create the corresponding directory and grant permissions as follows:

Mkdir/{backup, www}

Chown root: root-R/backup/

Chown root: root-R/www/

The above is the configuration of the rsync server.

7.2 rsyncClient Configuration

The configuration of the rsync client is significantly different from that of the server.

After the rsync service is installed on the client, you do not need to start the rsync service. We only need to create the password file required by the Verification Code user to connect to the rsync server on the client.

The password in the password file must correspond to the password in the password file on the rsync server, and must also match the user in the authentication module in the rsyncd. conf configuration file of the rsync server.

Of course, this password file can also be created without being created. When executing the rsync operation, we need to enter the password of the corresponding user of rsync. As follows:

Echo "apache">/etc/rsyncd. password

Chmod 600/etc/rsyncd. password

Note: Only one user's password can be stored in the password file, and the file must be set to 600 in the same way as the password file on the rsync server.

The above is the configuration of the rsync client.

8. rsync startup and Shutdown

After configuring the rsync server and client, we can start the rsync server and disable the rsync service.

8.1Start rsync

As we mentioned earlier, the rsync-daemon authentication method requires that the rsync service be installed on both the server and client, and you only need to start rsync on the rsync server and configure the rsync configuration file, the client does not start the rsync service.

Here, we will only explain how to start rsync-daemon authentication. We will not explain how to use ssh for rsync authentication. If you want to know more about this, you can refer to the six working modes of rsync in section 6.2.

For rsync installed in the source code method, we can use rsync-daemon to start rsync. As follows:

Echo PATH = $ PATH:/usr/local/bin/>/etc/profile

Source/etc/profile

Rsync -- daemon

Ps aux | grep rsync

Netstat-tunlp | grep 873

Note: In the preceding command line, only rsync -- daemon is used to start rsync. When the command is started, the/etc/rsyncd. conf file is loaded by default.

Therefore, if the rsync configuration file is not in the/etc directory or the rsync configuration file name is not rsyncd. conf, We need to manually add the complete path of the configuration file when starting the rsync service. The method is as follows:

Rsync -- daemon -- config =/etc/rsyncd. conf

You can use rsync -- daemon -- help to view how to load the configuration file. As follows:

Rsync -- daemon -- help

If rsync is installed in RPM mode, we have explained in Chapter 5.2 rsync installation in RPM mode. You can start rsync by starting the xinetd service. As follows:

/Etc/init. d/xinetd start

The preceding figure shows how to disable the rsync service.

8.2Disable rsync

To disable the rsync service, if the source code is used for installation, we can directly disable it using the pkill rsync command. As follows:

Pkill rsync

If the installation is based on yum, we can disable the xinetd service to disable the rsync service. As follows:

/Etc/init. d/xinetd stop

In addition to the above two methods, we can also disable the rsync service in a more violent way. As follows:

Kill-9 6780

IX. rsync synchronization instance

After the rsync service starts normally, we need to switch to the client to actually synchronize a file.

The rsync server is 192.168.199.247 and the rsync client is 192.168.199.248.

First, create an ilanni247 file under the/www/directory of the www module of the rsync server 192.168.199.247. The content is as follows:

Ifconfig eth1 | grep "inet addr" | awk '{print $2}' | cut-d:-f2

Vi/www/ilanni247

This is server 247 www.

Create a www directory under/root of rsync client 192.168.199.248, and create a file ilanni248 under this directory. The content is as follows:

Ifconfig eth1 | grep "inet addr" | awk '{print $2}' | cut-d:-f2

Mkdir www

Vi/www/ilanni248

This is client 248 www.

9.1Push 248 files to 247

To push the rsync client 192.168.199.248 to the rsync server 192.168.199.247, we need to use the third command in the rsync HELP Command, as shown below:

Rsync [OPTION]... SRC [SRC]... [USER @] HOST: DEST

We have already introduced the role of this command, that is, synchronizing local files to a remote server.

The usage is as follows:

Rsync-avz/root/www/* apache@192.168.199.247: www -- password-file =/etc/rsyncd. password

This command synchronizes all files in the/root/www directory to the www module on the 192.168.199.247 server.

To synchronize the/root/www/ilannidir directory to the www module on the 192.168.199.247 server, run the following command:

Rsync-azv/root/www/ilannidir apache@192.168.199.247: www -- password-file =/etc/rsyncd. password

Or:

Rsync-azv/root/www/ilannidir apache@192.168.199.247: www -- password-file =/etc/rsyncd. password

We can see that the local file and directory have been synchronized to the rsync server.

9.2Pull the 247 file to 248

To pull the rsync server 192.168.199.247 file to the rsync client 192.168.199.248, we need to use the sixth command in the rsync HELP Command, as shown below:

Rsync [OPTION]... [USER @] HOST: SRC [DEST]

We have already introduced the role of this command, that is, synchronizing the files on the remote machine to the local machine.

The usage is as follows:

Apache@192.168.199.247: www -- password-file =/etc/rsyncd. password/root/www/

This command is used to synchronize all the files in the www module of the 192.168.199.247 server to the/root/www/directory of the local machine.