Slime: Synchronization of Linux Files Rsync Learning (i)

This article by show according to Lin Feng to provide friendship sponsorship, starting in the mud row world.

Just have a few days to start learning Linux file synchronization software rsync, when learning rsync, we can be divided into the following several steps:

1. What is rsync?

2. How rsync Works

3. Rsync advantages

4. Rsync authentication method

5. Install Rsync

6, the rsync command parameter detailed

7. Configuring rsync

8, the start and shutdown of rsync

9. Rsync Synchronous Instance

Description: Both the rsync server and the client use the OS: CentOS 6.5 64bit.

First, what is rsync

Before we begin the formal learning of rsync, let's first answer this question: what is rsync.

rsync (remote synchronize) is a Liunx/unix data synchronization tool. It can quickly synchronize files and directories between multiple hosts via Lan/wan, and leverage the rsync algorithm (differential encoding) to reduce data transmission.

The rsync algorithm does not transmit all the time, but transmits only the different parts of the two files, so its transmission speed is quite fast.

In addition, rsync can copy, display directory attributes, and copy files, and optionally compress and recursively copy.

Second, the working principle of rsync

1. The client construction filelist,filelist contains all the file information that needs to be synchronized with the server to Name->id (the ID is used to uniquely represent the file such as MD5).

2. The client sends filelist to the server.

3, the server on the Rsync processing client sent over the filelist, build a new newfilelist. It deletes the file information pair that already exists on the server based on the MD5 value comparison, leaving only files that do not exist or change on the server.

4, the client to get the server sent over the newfilelist, and then transfer the files in the newfilelist to the server.

Iii. Advantages of Rsync

Rsync has several advantages:

1) can be mirrored to save the entire directory tree and file system.

2) It is easy to maintain the original file permissions, time, soft and hard connection.

3) installation without special privileges.

4) have optimized process and relatively high efficiency of file transfer.

5) You can use the shell (rsh, SSH) mode to transfer files.

6) Support anonymous operation.

7) compared with SCP, rsync transmission speed is definitely far more than the SCP transmission speed.

We often use rsync and SCP in the LAN to transfer a large number of MySQL database files, found that rsync transfer files at least 20 times times faster than SCP.

So if you need to transfer massive amounts of data between Liunx/unix servers, it is recommended that you choose rsync for transmission.

Four, the Rsync authentication method

Rsync has two commonly used authentication methods, one is the Rsync-daemon way, the other is the SSH mode. In the normal use of the process, we use the most is the Rsync-daemon way.

Note: When using rsync, both the server and the client must have the Rsync program installed .

4.1 Rsync-daemon Certifications

Rsync listens to TCP port 873 by default in Rsync-daemon authentication mode.

The Rsync-daemon authentication method is the main authentication method of Rsync, which is also a kind of authentication method that we often use. And it is only in this mode that rsync can write the password to a file.

Note: The Rsync-daemon authentication method requires both the server and the client to install the Rsync service, and only the rsync server is required to start rsync and configure the rsync configuration file. Client startup does not start the Rsync service, and does not affect the normal operation of synchronization.

4.2 ssh Certifications

rsync in SSH authentication mode, can be authenticated by the system user, that is, on rsync via SSH tunnel transmission, similar to the SCP tool. The synchronization operation is not limited to the synchronization folder defined in Rsync.

Note: The SSH authentication method does not require the server and client configuration rsync configuration file , the Rsync service needs to be installed on both sides, and there is no need for both sides to start rsync.

If the rsync server SSH is the standard port, then rsync is used as follows:

rsync-avz/root/test [Email protected]:/root/

If the rsync server SSH is a non-standard port, you can specify the port via the-e parameter of rsync. Use the following methods:

Rsync-avz/root/test-e ' ssh-p1234 ' [email protected]:/root/

V. Installation of rsync

To install rsync, we can be divided into two ways: source mode installation and RPM mode installation.

Note: The Rsync software is the same package both on the server side and on the client.

Below we will explain one by one, the two ways of installation method.

5.1 Source Mode installation

Source mode to install rsync, we need to download the corresponding installation package on their official website. Rsync official website: rsync.samba.org. As follows:

We use wget to download the following:

wget http://rsync.samba.org/ftp/rsync/rsync-3.1.1.tar.gz

After the installation package has been downloaded, we begin to unzip and install it. As follows:

TAR-XF rsync-3.1.1.tar.gz

./configure

Make &&make Install

Note: When installing rsync in the source code, the GCC library files required for compilation are installed as early as possible.

The rsync source installation will install rsync into the/usr/local/directory by default, and we can see through the relevant help. As follows:

./configure--help

After rsync has been installed, we can check the use of rsync for help. We need to switch to the/usr/local/bin directory at this time, as follows:

/usr/local/bin

Rsync-h

Source mode installation of rsync, to enable it to start the word. We can put the boot command directly into the rc.local file. As follows:

echo "/usr/local/bin/rsync--daemon--config=/etc/rsyncd.conf" >>/etc/rc.local

Can also be written as:

echo "/usr/local/bin/rsync--daemon" >>/etc/rc.local

Cat/etc/rc.local

5.2 RPM Mode installation

The RPM method of installing rsync is simple and can be installed directly using YUM. As follows:

Yum-y Install Rsync

Check the location of the rsync installation as follows:

RPM-QL rsync-3.0.6-12.el6.x86_64

Rsync, which can be installed in RPM mode, installs rsync into the/usr/bin directory and also generates a/etc/xinetd.d/rsync file.

Review the help documentation that was generated during the installation, as follows:

More/usr/share/doc/rsync-3.0.6/readme

After the installation is complete, we can look at the directory where the rsync command is located and the help of rsync. As follows:

Which rsync

Rsync–-help

It is clear that rsync is actually installed in the/usr/bin directory.

RPM installation of rsync, in order to enable it to start, we can also let it be installed with the source code to boot the same way.

In addition, there is a way to enable it to boot. If you look at it, you will find the/etc/xinetd.d/rsync file that rsync generated when it was installed.

Rsync runs the Rsync service by default in xinetd mode under CentOS. So the rpm way to install the word rsync, we just need to start the Xinet service.

However, before starting the XINETD service, we also need to configure the file/etc/xinetd.d/rsync as follows:

Vi/etc/xinetd.d/rsync

Once configured, we also need to install the XINETD package, otherwise the XINETD service cannot be started. As follows:

Yum-y Install xinetd

/etc/init.d/xinetd start

Chkconfig xinetd on

NETSTAT-TUNLP |grep 873

Six, the rsync command parameter detailed

After Rsync is installed, let's look at the Help for the rsync command below:

Rsync--help

6.1 rsync several important parameters

We will introduce a few of the frequently used parameters here:-v,-a,-z, the specific instructions for these parameters are described in this article "Slime:" to "rsync command parameters in detail."

-V,--verbose verbose mode output.

-A,--archive archive mode, which means that files are transferred recursively and that all file attributes remain intact.

-Z,--compress compresses the files that are backed up when they are transferred.

--delete: Delete files that are present in DST and not in SRC.

6.2 rsync Six modes of Operation

In addition to these several parameters, we also noted the seven command formats for rsync:

1) rsync [OPTION] ... src [src] ... DEST

2) rsync [OPTION] ... src [src] ... [[email protected]] Host:dest

3) rsync [OPTION] ... src [src] ... [[email protected]] HOST::D EST

4) rsync [OPTION] ... src [src] ... rsync://[[email protected]]host[:P ort]/dest

5) rsync [OPTION] ... [[email protected]] HOST:SRC [DEST]

6) rsync [OPTION] ... [[email protected]] HOST::SRC [DEST]

7) rsync [OPTION] ... rsync://[[email protected]]host[:P ort]/src [DEST]

These seven command formats represent rsync six different modes of operation, of which the fourth and seventh modes do not differ greatly. In these modes the third and sixth patterns are often used, especially the third model.

These six modes can be differentiated by two words in general: push, pull.

The push is to execute the rsync command on the client to push the files that the client needs to synchronize to the server.

Pull also executes the rsync command on the client to pull the files on the server locally.

Note: Both push and pull, the rsync command is executed on the client, but the command is in a different format.

1) rsync [OPTION] ... src [src] ... DEST

Synchronize local files, synchronizing from one directory to another. such as: Rsync-avz/data/backup, means to synchronize the local/data directory files to the local/backup directory.

2) rsync [OPTION] ... src [src] ... [[email protected]] Host:dest

Use a remote shell program (such as rsh, SSH) to synchronize the local files to the remote machine. This is a way of pushing. such as: rsync-avz/data [email protected]:/backup, the local/data directory to synchronize the files to the server 192.168.199.247/backup directory.

3) rsync [OPTION] ... src [src] ... [[email protected]] HOST::D EST

Synchronize the local files to the remote server , where dest represents the authentication module name of the rsync server. This is a way of pushing. such as: rsync-avz/data [email Protected]::backup--password-file=/etc/rsyncd.password, indicating the local/ The files in the data directory are synchronized to the server 192.168.199.247 under path under the Backup module.

4) rsync [OPTION] ... src [src] ... rsync://[[email protected]]host[:P ort]/dest

Lists a list of files for a remote machine. This is similar to the LS command, but only if the local machine information is omitted from the command. For example: Rsync-v rsync://[email protected]/backup, which indicates the contents of the/backup directory of the server 192.168.199.247 listed in this machine, as follows:

5) rsync [OPTION] ... [[email protected]] HOST:SRC [DEST]

Synchronizing files of remote machines locally, this is a pull method . such as: rsync-avz [email Protected]:/backup/data, the 192.168.199.247/backup directory to synchronize files to the local/data directory.

6) rsync [OPTION] ... [[email protected]] HOST::SRC [DEST]

Synchronizing files of remote machines locally, this is a pull method . such as: rsync-avz [email Protected]::backup--password-file=/etc/rsyncd.password/data, This indicates that the files under the path path of the 192.168.199.247 backup module are synchronized to the local/data directory.

Vii. Configuring Rsync

After Rsync is installed, we need to configure rsync. And for the rsync server, the most important and most complex is its configuration.

The configuration of rsync is divided into server side and client, below we explain one by one separately.

7.1 rsync server-side configuration

Rsync server requires two configuration files: rsyncd.conf, Rsyncd.password.

Where rsyncd.conf is stored by default in the/etc/directory, and it is also the main configuration file for the rsync server. This file configures the control authentication, access, logging, and so on for the rsync server. The Rsyncd.password is primarily used to store rsync usernames and passwords.

However, the above two configuration files will not be generated after rsync is installed, and we need to create them manually.

The contents of the rsyncd.conf configuration file are as follows:

Cat/etc/rsyncd.conf

UID = root

GID = root

User Chroot = No

Max connections = 200

Timeout = 600

PID file =/var/run/rsyncd.pid

Lock file =/var/run/rsyncd.lock

Log file =/var/run/rsyncd.log

[Backup]

Path =/backup/

Ignore errors

Read Only = no

List = no

Hosts allow = 192.168.199.0/255.255.255.0

Auth users = Test

Secrets file =/etc/rsyncd.password

[WWW]

Path =/www/

Ignore errors

Read Only = no

List = no

Hosts allow = 192.168.199.0/255.255.255.0

Auth users = Apache

Secrets file =/etc/rsyncd.password

For a detailed explanation of the rsync configuration file, see the next article, "Slime: rsync configuration File".

rsync The default is to run under the nobody user, but we do not bother about various permissions for later synchronization, let it run under the root user.

After the configuration file is created, we create the password file. As follows:

echo "Test:test" >>/etc/rsyncd.password

echo "Apache:apache" >>/etc/rsyncd.password

Cat/etc/rsyncd.password

Note the user and password in the password file can be customized, and the user does not necessarily have to exist on the system user in the server.

After the password file is created, to set its permissions to 600, you will be prompted to verify the error later when synchronizing. As follows:

chmod 600/etc/rsyncd.password

An rsync configuration file can contain multiple authentication modules, while a password file can also hold multiple users and their corresponding passwords. Each of these authentication modules can correspond to a different client.

Once the above two files have been created, we will create their corresponding directories and authorize them as follows:

mkdir/{backup,www}

Chown root:root–r/backup/

Chown root:root–r/www/

The above is the Rsync server-side configuration.

7.2 rsync Client Configuration

The rsync client configuration is still relatively large compared to the server side.

After the client has installed the Rsync service, it is not necessary to start the rsync service. We only need the password file that the CAPTCHA user needs when creating the client connection to the Rsync server.

The password in this password file corresponds to the password in the password file on the rsync server and is also matched to the user in the authentication module in the rsync server rsyncd.conf configuration file.

Of course, the password file is not created and is also possible, not created. When we perform the rsync operation, we need to enter the password of the corresponding user of rsync. As follows:

echo "Apache" >>/etc/rsyncd.password

chmod 600/etc/rsyncd.password

Note: This password file can only hold one user's password, and the file will need to be set to 600 as well as the rsync server-side password file.

The above is the configuration of the Rsync client.

Viii. start-up and shutdown of rsync

After the rsync server and client are configured, we can start the rsync server and also turn off the rsync service.

8.1 start the rsync service

In the previous we talked about the Rsync-daemon authentication method, requires both the server and the client to install the Rsync service, and only requires Rsync server side to start rsync, while configuring the rsync configuration file, the client start does not start the rsync service.

Here we only explain the start of the Rsync-daemon certification method. For rsync use SSH authentication, we do not explain here, if you want to know the children's shoes, you can see the 6.2 section of the Rsync six mode of operation.

If it is a source-style installation of rsync, we can use Rsync–daemon to start rsync. As follows:

echo path= $PATH:/usr/local/bin/>>/etc/profile

Source/etc/profile

Rsync--daemon

PS aux |grep rsync

NETSTAT-TUNLP |grep 873

Note: In the above command line, only rsync--daemon is the command that initiates rsync. And when the command starts, the/etc/rsyncd.conf file is loaded by default.

So if the rsync configuration file is not in the/etc directory or the rsync configuration file name is not rsyncd.conf, then when we start the rsync service, we need to manually add the full path to the configuration file. Here's how:

Rsync--daemon--config=/etc/rsyncd.conf

For information on how to load the configuration file, we can view it via Rsync--daemon--help. As follows:

Rsync--daemon--help

In the case of rsync installed in RPM mode, we have already explained in the 5.2 chapter RPM installation rsync and can start rsync by starting the XINETD service. As follows:

/etc/init.d/xinetd start

This is the start of the Rsync service, and we'll start by describing how to turn off the rsync service.

8.2 Turn off rsync services

To turn off the rsync service, if it is installed in source mode, we can close it directly via the Pkill rsync command. As follows:

Pkill rsync

If the installation is done through Yum, we can turn off the rsync service by turning off the xinetd service. As follows:

/etc/init.d/xinetd stop

In addition to the above two methods, we can also close the Rsync service by comparing violent methods. As follows:

Kill-9 6780

Nine, rsync synchronous instance

After the Rsync service starts normally, we switch to the client to actually synchronize a file.

Now the rsync server is the 192.168.199.247,rsync client for 192.168.199.248.

First in the rsync server 192.168.199.247 www module under the/www/directory under the new file ilanni247, the contents are as follows:

Ifconfig eth1|grep "inet addr" |awk ' {print $} ' |cut-d:-f2

vi/www/ilanni247

This is server 247 www.

Then create a new www directory under the/root of the rsync client 192.168.199.248, creating a file ilanni248 in this directory with the following contents:

Ifconfig eth1|grep "inet addr" |awk ' {print $} ' |cut-d:-f2

mkdir www

vi/www/ilanni248

This is client 248 www.

9.1 push 248 files to 247.

To push the rsync client 192.168.199.248 file to the rsync server 192.168.199.247, we need to use the third command in the Rsync help command, as follows:

rsync [OPTION] ... src [src] ... [[email protected]] HOST::D EST

The role of this command, as we have already described earlier, is to synchronize the local files to the remote server.

The specific use method is as follows:

rsync-avz/root/www/* [Email protected]::www--password-file=/etc/rsyncd.password

The purpose of this command is to synchronize all files under the/root/www directory to the 192.168.199.247 server's WWW module.

If you want to synchronize the/root/www/ilannidir directory itself to the 192.168.199.247 server's WWW module, the command is as follows:

Rsync-azv/root/www/ilannidir [Email protected]::www--password-file=/etc/rsyncd.password

Or:

Rsync-azv/root/www/ilannidir [Email protected]::www--password-file=/etc/rsyncd.password

With this, we can see that the files and directories have been synced to the rsync server locally.

9.2 Pull 247 files to 248.

To pull the rsync server 192.168.199.247 file to the Rsync client 192.168.199.248, we need to use the Sixth command in the Rsync help command, as follows:

rsync [OPTION] ... [[email protected]] HOST::SRC [DEST]

The role of this command, as we have already described earlier, is to synchronize the remote machine's files locally.

The specific use method is as follows:

Rsync-avz [Email protected]::www--password-file=/etc/rsyncd.password/root/www/

The purpose of this command is to synchronize all files under the WWW module of the 192.168.199.247 server to the/root/www/directory of this machine.

Slime: Synchronization of Linux Files Rsync Learning (i)