Real-time data synchronization using rsync + inotify in Linux

This site has been missing a similar tutorial. I will add it today. Rsync is a remote data synchronization tool. You can refer to the previous article Rsync remote data synchronization commands in linux and how to use them. however, after the data is updated, the data still needs to be manually synchronized and cannot be automated. You can configure rsync + innotify to synchronize data in real time.

Environment preparation: three centos servers
SOURCE Data Server: 192.168.1.8
Target server: 192.168.1.9 192.168.1.10
Objective: to automatically synchronize/home/21 yunwei of the source server to the target server.
To test the function, disable selinux and iptables (iptables can also allow port 873 without closing)
1. Configure two target servers
1. Install rsync and xinetd. In centos, rsync is managed through xinetd.

Yum install rsync xinetd-y
Edit vi/etc/xinetd. d/rsync to set rsync to start, that is, disable = no. Yes is disabled by default.
After modification, start xinetd:/etc/init. d/xinetd start
Netstat-tnlp | grep 873 if the xinetd service is available, it indicates that the service is successfully started.
2. Create the rsyncd. conf configuration file and modify the parameters.
Vi/etc/rsyncd. conf

Log file =/var/log/rsyncd. log
Pidfile =/var/run/rsyncd. pid
Lock file =/var/run/rsync. lock
Secrets file =/etc/rsync. pass
Motd file =/etc/rsyncd. Motd
 
[Home_21yunwei]
Path =/home/21 yunwei/
Comment = home_21yunwei
Uid = root
Gid = root
Port = 873
Use chroot = no
Read only = no
List = no
Max connections = 200
Timeout = 600
Auth users = 21 yunwei
Hosts allow = 192.168.1.8
Hosts deny = 192.168.1.200
Here are the parameter descriptions:

Log file =/var/log/rsyncd. log # log file location. This file is automatically generated after rsync is started. You do not need to create it in advance.
Pidfile =/var/run/rsyncd. pid # storage location of the pid file
Lock file =/var/run/rsync. lock # lock files supporting the max connections parameter
Secrets file =/etc/rsync. pass # The user authentication configuration file, which stores the user name and password. This file will be created later.
Motd file =/etc/rsyncd. Motd # Welcome to the file location on the page when rsync is started (custom file content)
 
[Home_21yunwei] # Custom name
Path =/home/21 yunwei/# rsync server data directory path
Comment = home_21yunwei # The module name is the same as the custom name of [home_www.osyunwei.com]
Uid = root # set rsync operation permission to root
Gid = root # set rsync operation permission to root
Port = 873 # default port
Use chroot = no # The default value is true. Change it to no to add a backup for the soft connection to the directory file.
Read only = no # Set The rsync server file to read and write permissions
List = no # The rsync server resource list is not displayed.
Max connections = 200 # maximum number of connections
Timeout = 600 # set the timeout value
Auth users = 21 yunwei # User name for data synchronization. Multiple user names can be set and separated by commas (,).
Hosts allow = 192.168.1.8 # IP address of the client that allows data synchronization. You can set multiple IP addresses separated by commas (,).
Hosts deny = 192.168.1.200 # IP address of the client that prohibits data synchronization. You can set multiple IP addresses separated by commas (,).
3. Create a user authentication file

Vi/etc/rsync. pass
21 yunwei: 123456 # Format, user name: password, you can set multiple, one user name per line: password
4. Set file permissions
Chmod 600/etc/rsyncd. conf
Chmod 600/etc/rsync. pass
If you do not set this permission, the following error occurs: password file must not be other-accessible.

Restart the xinetd service:
/Etc/init. d/xinetd restart

2. Source data server configuration

1. Install rsync and xinetd.
Basically, the installation of the target server is the same. After installation, set the service to start. Start the xinetd service.
2. Create an authentication password file
Vi/etc/passwd.txt write:
123456 # Password

Set permissions:
Chmod 600/etc/passwd.txt
If no permission is set, the following error occurs: password file must not be other-accessible.
3. Test the data synchronization between the source server 192.168.1.8 and the target server.
Objective: to synchronize files under/home/21yunwei to two target servers.
We run the test after touch/home/21 yunwei {1 .. 10}. php:

Rsync-avz -- port = 873 -- delete/home/21 yunwei/21yunwei@192.168.1.9: home_21yunwei -- password-file =/etc/passwd.txt
Rsync-avz -- port = 873 -- delete/home/21 yunwei/21yunwei@192.168.1.10: home_21yunwei -- password-file =/etc/passwd.txt
Test results:

[Root @ kvmserver home] # rsync-avz -- port = 873 -- delete/home/21 yunwei/21yunwei@192.168.1.10: home_21yunwei -- password-file =/etc/passwd.txt
 
Sending incremental file list
./
1. php
10. php
2. php
3. php
4. php
5. php
6. php
7. php
8. php
9. php
 
Sent 557 bytes encoded Ed 206 bytes 1526.00 bytes/sec
Total size is 4 speedup is 0.01
View the Directory of the 10 target servers:

[Root @ web10 home] # ll 21 yunwei/
Total 8
-Rw-r -- 1 root 0 Sep 1 :36 10.php
-Rw-r -- 1 root 4 Sep 1 1.php
-Rw-r -- 1 root 0 Sep 1 :36 2.php
-Rw-r -- 1 root 0 Sep 1 :36 3.php
-Rw-r -- 1 root 0 Sep 1 :36 4.php
-Rw-r -- 1 root 0 Sep 1 :36 5.php
-Rw-r -- 1 root 0 Sep 1 :36 6.php
-Rw-r -- 1 root 0 Sep 1 :36 7.php
-Rw-r -- 1 root 0 Sep 1 :36 8.php
-Rw-r -- 1 root 0 Sep 1 :36 9.php
The two servers have been tested, and the performance test has no problem in synchronizing data.

3. Install Inotify-tools on the source server to trigger rsync in real time.

1. Check whether the server kernel supports inotify.
Ll/proc/sys/fs/inotify

[Root @ kvmserver home] # ll/proc/sys/fs/inotify
Total 0
-Rw-r -- 1 root 0 Sep 1 17:20 max_queued_events
-Rw-r -- 1 root 0 Sep 1 17:20 max_user_instances
-Rw-r -- 1 root 0 Sep 1 17:20 max_user_watches
If the preceding content is displayed, innotify is supported.
2. Install inotify-tools
Download the http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz package and decompress it, compile and install it.

Tar zxvf inotify-tools-3.14.tar.gz
Cd inotify-tools-3.14
./Configure -- prefix =/usr/local/inotify
Make
Make install
3. Set environment variables and create soft links.

Echo "PATH =/usr/local/inotify/bin: $ PATH">/etc/profile. d/inotify. sh
Source/etc/profile. d/inotify. sh
Echo "/usr/local/inotify/lib">/etc/ld. so. conf. d/inotify. conf
Ln-s/usr/local/inotify/include/usr/include/inotify
4. Modify the default inotify parameter.
Modify parameters:

Sysctl-w fs. inotify. max_queued_events = "99999999"
Sysctl-w fs. inotify. max_user_watches = "99999999"
Sysctl-w fs. inotify. max_user_instances = "65535"
Edit vi/etc/sysctl. conf

Fs. inotify. max_queued_events = 99999999
Fs. inotify. max_user_watches = 99999999
Fs. inotify. max_user_instances = 65535
Parameter description:
Max_queued_events:
The maximum length of the inotify Queue. If the value is too small, the "** Event Queue Overflow **" error will occur, resulting in inaccurate monitoring files.

Max_user_watches:
Directory of the file to be synchronized. Avoid the failure caused by this value being too small, especially when there are many small file directories.
Max_user_instances:
The maximum value for creating inotify instances for each user.

5. Create a script to trigger rsync in real time for synchronization.

#! /Bin/sh
 
Srcdir =/home/21 yunwei/
Dstdir = home_21yunwei
Excludedir =/usr/local/inotify/exclude. list
Rsyncuser = 21 yunwei
Rsyncpassdir =/etc/passwd.txt
Dstip = "192.168.1.9 192.168.1.10"
 
For ip in $ dstip
Do
Rsync-avH -- port = 873 -- progress -- delete -- exclude-from = $ excludedir $ srcdir $ rsyncuser @ $ ip: $ dstdir -- password-file = $ rsyncpassdir
Done
 
/Usr/local/inotify/bin/inotifywait-mrq -- timefmt '% d/% m/% y % H: % M' -- format' % T % w % f % e'-e close_write, modify, delete, create, attrib, move $ srcdir | while read file
Do
For ip in $ dstip
Do
Rsync-avH -- port = 873 -- progress -- delete -- exclude-from = $ excludedir $ srcdir $ rsyncuser @ $ ip: $ dstdir -- password-file = $ rsyncpassdir
Echo "$ {file} was rsynced">/tmp/rsync. log 2> & 1
Done
Done
Script parameter description:

Srcdir =/home/21 yunwei/# synchronization Directory of the source server
Dstdir = home_21yunwei # Name of The rsync synchronization directory module on the target server
Excludedir =/usr/local/inotify/exclude. list
# Directories that do not need to be synchronized. If there are multiple directories and one directory is written in each row, the path relative to the synchronization module is used;
# For example, you do not need to synchronize the directory under the/home/21 yunwei/directory and the b1 directory under the B Directory. The exclude. list file can be written in this way.
A/
B/b1/

Rsyncuser = 21 yunwei # target server rsync user name
Rsyncpassdir =/etc/passwd.txt # the target server rsync synchronizes the user's password to the storage path of the source server
Dstip = "192.168.1.9 192.168.1.10 & Prime; # ip address of the target server. Separate multiple ip addresses with spaces.
/Tmp/rsync. log # script running log records

6. Set the script to automatically execute upon startup

Vi/etc/rc. d/rc. local
/Bin/bash/usr/local/inotify/rsync. sh & amp; # set automatic startup and run scripts in the background
Restart the server.
7. Test if the inotify trigger rsync synchronization script in real time is normal.
We create a file in/home/21 yunwei on the source server:

[Root @ kvmserver 21 yunwei] # echo "test inotify"> 21yunwei. php
[Root @ kvmserver 21 yunwei] #
Sending incremental file list
./
21yunwei. php
13 100% 0.00kB/s 0:00:00 (xfer #1, to-check = 8/14)
 
Sent 268 bytes encoded Ed 30 bytes 596.00 bytes/sec
Total size is 17 speedup is 0.06
 
Sending incremental file list
21yunwei. php
13 100% 0.00kB/s 0:00:00 (xfer #1, to-check = 8/14)
Then go to the 192.168.1.9 and 192.168.1.10 directories to view the two files. The same is true for deletion:

[Root @ web10 21 yunwei] # ll
Total 12
-Rw-r -- 1 root 0 Sep 1 :36 10.php
-Rw-r -- 1 root 4 Sep 1 1.php
-Rw-r -- 1 root 0 Sep 1 :49 1.txt
-Rw-r -- 1 root 13 Sep 1 :51 21yunwei. php
In another test, we updated the svn content to/home/21 yunwei and checked whether the web 9 and web10 nodes also changed:

[Root @ kvmserver 21 yunwei] # svn co svn: // 59.46.x.x/21 yunweinew/home/21 yunwei/-- username = 21 yunwei -- password = xxxxxx
Go to web9 and web10 and find that the newly updated data is also in progress.
Common inotify parameters:

Inotify parameters
-M: Keep Listening
-R is used to recursively View directories.
-Q indicates the output event.
-E create, move, delete, modify, and attrib refer to the "listening for creating a mobile delete write permission" event.
So far, inotify + rsync settings are complete.