Rsync and inotify for seamless data synchronization

Rsync is a remote differential data synchronization tool that copies one copy of data to another machine. Its synchronization feature is that, in the entire file domain, only changed files are synchronized according to the file pattern calculated based on The Rsync algorithm, but when synchronizing the entire domain, it will calculate the features of all files in the entire domain, so that it takes a lot of time to compute the file features when there is too much file data. Without other software, rsync cannot implement real-time data synchronization. Rsync can synchronize directories or the entire file system. secure transmission may be achieved through SSH encryption. Because only differential data is synchronized, Rsync is highly efficient and supports anonymous transmission and other features. Rsync is divided into the server and the client. As the server, you can wait for the client to push or pull data at any time. When Rsync is used as the server, it depends on the super daemon xinetd to listen to port tcp873. Therefore, xinetd must be installed.

If the system does not install rsync, go to the http://rsync.samba.org/download source code compilation, or use Yum-y install rsync for installation.

Rsync command used by the rsync Client

Syntax format

Pull file: rsync [option...] [[email protected] HOST: SRC... [DEST]

Rsync [option...] rsync: // [[email protected] host [: Port]/src... [DEST]

Push file: rsync [option...] SRC... [[email protected] HOST: dest

Rsync [option...] SRC... rsync: // [[email protected] host [: Port]/dest

Common options

-N: Test synchronization. Only tests do not actually synchronize data.

-V: displays the synchronization process.

-C: enable the verification function.

-R: recursive replication (if the directory is synchronized, do not write/behind the directory /. If write/is used to synchronize files in the directory)

-A: Archive synchronization. After data synchronization, the original attributes of the file are retained.

-P: Keep all permissions on the file.

-T: Specifies the timestamp of the object to be retained.

-L: Reserved symbolic connections

-O: owner of the reserved File

-G: Group of objects to be retained

-D: Keep the device file.

-E ssh: specifies that the SSH protocol is used for encrypted transmission.

-Z: compressed data transmission during transmission

-- Progress: displays the transmission progress bar.

-- Stats: displays how data is compressed and transmitted.

-- Delete: Delete more than files

-Q: Stop the error message.

Rsync server configuration file/etc/rsyncd. conf. The first section defines rsync as a feature for running services. The second part defines the features of the rsync module. The module defines the path used locally as the storage directory and the working features of this module. This file is not provided by default and needs to be created manually.

# The global settings global definition segment can only have one.

Uid = nobody the process runs as that user

Gid = user group for which the nobody process runs

Use chroot = No whether the chroot function is supported when Rsync is run

Max connections = 5 Max number of clients allowed for simultaneous requests

Strict modes = yes whether strict mode is enabled

PID file =/var/run/rsyncd. PID process ID file location

Log File =/var/log/rsyncd. Log location

# Directory to be synced is used to define synchronization directories. There can be multiple directories.

[Tools]

Path =/Data Storage synchronization Data Location

Ignore errors = yes whether to ignore the error during Data Transmission

Read Only = No whether it is read-only

Write only = No indicates whether to write only. If you write only, you cannot view it.

Hosts allow = 172.16.0.0/16 defines the client whitelist

Hosts deny = * defines the client that is not allowed to access the local machine by default. * indicates all.

List = false indicates that an example file is allowed. If yes, the value is true.

Uid = root specify the user identity used for this sharing

Gid = root specifies the user group ID used to run the shared operation.

Auth users = user1, user2 specifies users that can be synchronized through rsync

Secrets file =/etc/rsyncd. passwd specifies to verify the user's file

When starting the service, you need to modify the configuration file so that xinetd can listen to the service when it is started. File in/etc/xinetd. d/rsync

Service rsync

{

Disable = No. Change "yes" to "no ".

Flags = IPv6

Socket_type = stream

Wait = No

User = root

Server =/usr/bin/rsync

Server_args = -- daemon

Log_on_failure + = userid

}

After saving and exiting, restart the xinetd service. If TCP port 873 is in the listening status, the service is started normally.

650) This. width = 650; "Title =" Clipboard "style =" border-top: 0px; border-Right: 0px; border-bottom: 0px; border-left: 0px; display: inline "border =" 0 "alt =" Clipboard "src =" http://img1.51cto.com/attachment/201408/22/1080241_14087414106ntD.png "" 954 "Height =" 291 "/>

Test on the client

========================================================== ======================================

Inotify usage

Inotify is a tool that monitors file systems in real time, including file creation, modification, deletion, and viewing. This feature requires Kernel support later than 2.6.13, including 2.6.13. Inotify-tools is a tool used to interact with inotify of the kernel. It can be used to monitor every action of a specified file in real time. Configured and used with the sh script to achieve the monitoring effect. The installation method is as follows.

# Wget http://nchc.dl.sourceforge.net/project/inotify-tools/inotify-tools/3.13/inotify-tools-3.13.tar.gz

# Tar-XF inotify-tools-3.13.tar.gz

# Cd inotify-tools-3.13

#./Configure the installation path is not specified here. After the default installation, the executable program is in/usr/local/bin.

# Make & make install

# Echo "fs. inotify. max_user_watches = 100000">/etc/sysctl. conf: set the maximum number of monitored files

# Sysctl-p takes effect for the specified Kernel Parameters

Inotify-tools has two execution files after installation

Inotifywait, used to monitor file changes

Inotifywatch is used to count the number of accesses to the file system and monitor the metadata of files.

Command Options

-M,-Monitor:If a task is received without exiting, it is executed indefinitely. The default action is to exit immediately after receiving a message.

-R,-recursive: monitor all subdirectories in a directory

-Q,-Quiet: specify a time. No details are output. Specify a second time. Except for fatal errors, no information is output. If this parameter is specified once, no detailed information is output. If the parameter is specified twice, no information is output except fatal errors.

-E,-Event: only listens to the specified event.

There are the following types of events:

650) This. width = 650; "Title =" Clipboard [1] "style =" border-top: 0px; border-Right: 0px; border-bottom: 0px; border-left: 0px; display: inline "border =" 0 "alt =" Clipboard [1] "src =" http://img1.51cto.com/attachment/201408/22/1080241_1408741410YHoa.png "" 690 "Height =" 504 "/>

========================================================== ======================================

Configure an rsync + inotify case

650) This. width = 650; "Title =" Clipboard [2] "style =" border-top: 0px; border-Right: 0px; border-bottom: 0px; border-left: 0px; display: inline "border =" 0 "alt =" Clipboard [2] "src =" http://img1.51cto.com/attachment/201408/22/1080241_1408741411EjcJ.png "" 619 "Height =" 459 "/>

1. Node 1 and node 2 are both rsync servers,

2. The distributor pushes the changed data to node 1 and node 2 in real time.

3. Like the environment of Node 1 and node 2, the module name is HTML and the shared directory is/data.

4. Transmission Encryption

Operation Process ---------------------------------------------------------------------

In order to synchronize data without entering a password, select public key authentication. Generate a key pair on the distributor.

# Ssh-keygen-t rsa no password is set here, so you can press enter all the way.

# Cd/root/. Ssh/

# Cat id_rsa.pub view and copy the Public Key File

650) This. width = 650; "Title =" Clipboard [3] "style =" border-top: 0px; border-Right: 0px; border-bottom: 0px; border-left: 0px; display: inline "border =" 0 "alt =" Clipboard [3] "src =" http://img1.51cto.com/attachment/201408/22/1080241_1408741412Wk8g.png "" 822 "Height =" 438 "/>

Copy the text in the red box to the/root/. Ssh/authorized_keys file on the two node servers. If no, manually create the file. After the copy is complete, edit the rsync configuration file for the two node servers as follows. If there is no rsync, you need to install it.

650) This. width = 650; "Title =" Clipboard [4] "style =" border-top: 0px; border-Right: 0px; border-bottom: 0px; border-left: 0px; display: inline "border =" 0 "alt =" Clipboard [4] "src =" http://img1.51cto.com/attachment/201408/22/1080241_1408741413nEsP.png "" 323 "Height =" 296 "/>

After the configuration file is complete, restart the xinetd service.

After the distributor is installed, edit the following script to automatically synchronize data.

# Vim/bin/inotify. sh:

Source/etc/rsync. inotify

/Usr/local/bin/inotifywait-mrq-e modify, delete, create, attrib/Data | while read d E F

Do

/Usr/bin/rsync-ahqz-e SSH -- delete $ {source}/$ {Host }:: $ dest

/Usr/bin/rsync-ahqz-e SSH -- delete $ {source}/$ {host1}: $ dest

Done

# Vim/etc/rsync. inotify provides the configuration file for the above script file. The content is as follows: the configuration file is provided to flexibly expand multiple nodes in the future.

Source =/Data

Host = 172.16.2.10

DeST = html

Host1 = 172.16.2.15

# Echo "/bin/inotify. Sh &">/etc/rc. d/rc. Local # enable the script to automatically start upon startup

#/Bin/inotrfy. Sh start script. Enter yes when synchronizing data for the first time. So let it run on the foreground first.

# Cp/bin/I */data copy Point file for testing,

Viewing on two nodes may cause a delay.

# Ls/Data

650) This. width = 650; "Title =" Clipboard [5] "style =" border-top: 0px; border-Right: 0px; border-bottom: 0px; border-left: 0px; display: inline "border =" 0 "alt =" Clipboard [5] "src =" http://img1.51cto.com/attachment/201408/22/1080241_1408741413MpPZ.png "" 615 "Height =" 48 "/>

This completes

Rsync and inotify for seamless data synchronization