Real-time data synchronization using rsync + inotify

Real-time data synchronization using rsync + inotify

As the scale of application systems continues to expand, rsync also puts forward better requirements on data security and reliability. In the high-end business system, rsync gradually exposes many shortcomings. First, during rsync data synchronization, You need to scan all files for comparison and transfer the data in a different amount. If the number of files reaches millions or even tens of millions, scanning all files will be very time-consuming. And the changes are often a small part, which is very inefficient. Secondly, rsync cannot monitor and synchronize data in real time. Although it can trigger Synchronization Through the Linux daemon, there will be a time difference between the two triggering actions, as a result, the data on the server and the client may be inconsistent and the data cannot be completely restored when the application fails. For the above reason, the combination of rsync and inotify has appeared!

1.1 Introduction to inotify
2.1 logical diagram of rsync + inotify Synchronization
3.1 Environment deployment
4.1 inotify-slave deployment
4.1.1 check whether rsync is installed
4.1.2 create an rsync user and Module Directory and change its user group
4.1.3 compile the rsync daemon configuration file/etc/rsyncd. conf
4.1.4 configure the virtual User Password File
4.1.5 start the rsync Service
4.1.6 push through inotify-master Test
5.1 inotify-master deployment
5.1.1 check whether the current system supports inotify
5.1.2 download the inotify source code package and compile and install it
5.1.3 detailed description of common parameters of inotify inotifywait commands
5.1.4 write a monitoring script and load it to the background for execution
5.1.5 real-time synchronization test
 
1.1 Introduction to inotify
Inotify is a powerful fine-grained and asynchronous file system event control mechanism. Linux Kernel has been supported by inotify since 2.6.13. inotify can be used to monitor various events such as adding, deleting, modifying, and moving events in the file system, third-party software can monitor various file changes in the file system, while inotify-tools is the software that implements monitoring.

--------------------------------------------------------------------------------
2.1 logical diagram of rsync + inotify Synchronization

--------------------------------------------------------------------------------
3.1 Environment deployment

Host Name	Host IP Address	System Version	System kernel version
Inotify-master	192.168.1.128	CentOS release 6.4 (Final)	2.6.32-358. el6.x86 _ 64
Inotify-slave	192.168.1.160	CentOS release 6.4 (Final)	2.6.32-358. el6.x86 _ 64

4.1 inotify-slave deployment
Here we deploy the rsync service and the rsync daemon working mode.
4.1.1 check whether rsync is installed

[Root @ inotify-slave ~] # Rpm-aq rsync
Rsync-3.0.6-9.el6.x86_64

4.1.2 create an rsync user and Module Directory and change its user group
[Root @ inotify-slave mail] # useradd rsync-s/sbin/nologin-M # Add rsync users
[Root @ inotify-slave mail] # grep rsync/etc/passwd
Rsync: x: 2004: 2004:/home/rsync:/sbin/nologin
[Root @ inotify-slave mail] # mkdir/backup # create a Module Directory in rsync daemon Working Mode
[Root @ inotify-slave mail] # ll-d/backup/
Drwxr-xr-x. 2 root 4096 April 22 14:13/backup/
[Root @ inotify-slave mail] # chown rsync. rsync/backup/# change the user group of the Module Directory
[Root @ inotify-slave mail] # ll-d/backup/
Drwxr-xr-x. 2 rsync 4096 14:13/backup/

4.1.3 compile the rsync daemon configuration file/etc/rsyncd. conf
[Root @ inotify-slave/] # cat/etc/rsyncd. conf
# Rsyncd. conf start ##
# Specify a user in the work (users must be specified)
Uid = rsync
Gid = rsync
# Equivalent to black hole. error locating
Use chroot = no
# How many clients can upload files simultaneously?
Max connections = 200
# Timeout
Timeout = 300
# Process number File
Pid file =/var/run/rsyncd. pid
# Log files
Lock file =/var/run/rsync. lock
# Log files
Log file =/var/log/rsyncd. log
# Module start
# This module corresponds to the push directory
# Use the module name as needed
[Backup]
# Directories to be synchronized
Path =/backup/
# Indicates the error is ignored.
Ignore errors
# Indicates that the network permission is writable (local control is truly writable)
Read only = false
# Set the IP address or prevent Synchronization
List = false
# Specify the allowed CIDR blocks
Hosts allow = 192.168.1.0/24
# Indicates that no link is rejected.
Hosts deny = 0.0.0.0/32
# Do not touch anything (default)
# Virtual user
Auth users = rsync_backup
# Virtual User Password File
Secrets file =/etc/rsync. password
# End of the configuration file
# Rsync_config ___________ end

4.1.4 configure the virtual User Password File
[Root @ inotify-slave/] # echo "rsync_backup: leesir">/etc/rsync. password
[Root @ inotify-slave/] # cat/etc/rsync. password
Rsync_backup: leesir # Note: rsync_backup is a virtual user, and leesir is the password of this virtual user.
[Root @ inotify-slave/] # chmod 600/etc/rsync. password # privilege escalation for the password file to improve security
[Root @ inotify-slave/] # ll/etc/rsync. password
-Rw -------. 1 root 20 April 22 14:20/etc/rsync. password

4.1.5 start the rsync Service
[Root @ inotify-slave/] # rsync -- daemon # Start the rsync Service
[Root @ inotify-slave/] # ps-ef | grep rsync
Root 14871 1 0? 00:00:00 rsync -- daemon
Root 14873 14788 0 00:00:00 pts/0 grep rsync
[Root @ inotify-slave/] # netstat-lnutp | grep rsync
Tcp 0 0 0.0.0.0: 873 0.0.0.0: * LISTEN 14871/rsync
Tcp 0 0: 873: * LISTEN 14871/rsync

4.1.6 push through inotify-master Test
Inotify-master configuration password file, test push
[Root @ inotify-master ~] # Echo "leesir">/etc/rsync. password
[Root @ inotify-master ~] # Cat/etc/rsync. password
Leesir # Note: you only need to write the password here. Remember.
[Root @ inotify-master ~] # Chmod 600/etc/rsync. password
[Root @ inotify-master ~] # Ll/etc/rsync. password
-Rw ------- 1 root 7 April 22 14:32/etc/rsync. password
[Root @ inotify-master ~] # Echo "hello leesir"> test.txt
[Root @ inotify-master ~] # Cat test.txt
Hello leesir
[Root @ inotify-master ~] # Rsync-avztest.txt rsync_backup@192.168.1.160: backup -- password-file =/etc/rsync. password
Sending incremental file list
Test.txt
Sent 82 bytes encoded ed 27 bytes 72.67 bytes/sec
Total size is 13 speedup is 0.12

Inotify-slave check:
[Root @ inotify-slave/] # ll/backup/
Total usage 4
-Rw-r --. 1 rsync 13 Aug 17 14:34 test.txt
[Root @ inotify-slave/] # cat/backup/test.txt
Hello leesir

-------------------------------------------------------------------------------
5.1 inotify-master deployment
Note:
Inotify is installed and executed by the rsync client.
-Synchronization restrictions in enterprise scenarios, restricted by network adapter, disk, bandwidth, etc.
5.1.1 check whether the current system supports inotify
[Root @ inotify-master ~] # Ll/proc/sys/fs/inotify/
Total usage 0
-Rw-r -- 1 root 0 April 22 14:56 max_queued_events
-Rw-r -- 1 root 0 April 22 14:56 max_user_instances
-Rw-r -- 1 root 0 April 22 14:56 max_user_watches
# The three files are supported.

Expansion:
/Proc/sys/fs/inotify/max_queued_evnets
Indicates the maximum number of events that can be queued in the inotify instance when inotify_init is called. Events that exceed this value are discarded, but the IN_Q_OVERFLOW event is triggered.
/Proc/sys/fs/inotify/max_user_instances
The maximum number of inotify instatnces that can be created by each real user ID.
/Proc/sys/fs/inotify/max_user_watches
The maximum number of directories that each inotify instatnces can monitor. If the number of monitored files is large, you need to increase the size of this value as appropriate.
Example: echo 30000000>/proc/sys/fs/inotify/max_user_watches
 
5.1.2 download the inotify source code package and compile and install it
[Root @ inotify-master tools] # wget http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz # download inotify source package
..................................
Root @ inotify-master tools] # ll inotify-tools-3.14.tar.gz
-Rw-r -- 1 root 358772 March 14 2010 inotify-tools-3.14.tar.gz
[Root @ inotify-master tools] # tar zxf inotify-tools-3.14.tar.gz
[Root @ inotify-master tools] # cd inotify-tools-3.14
[Root @ inotify-master inotify-tools-3.14] #./configure -- prefix =/usr/local/inotify-3.14 # configure inotify and specify the installation path to/usr/local/inotify-3.14
................................
[Root @ inotify-master inotify-tools-3.14] # make & make install
................................

5.1.3 detailed description of common parameters of inotify inotifywait commands
[Root @ inotify-master inotify-tools-3.14] # cd/usr/local/inotify-3.14/
[Root @ inotify-master inotify-3.14] #./bin/inotifywait -- help
-R | -- recursive Watch directories recursively. # recursive query directory
-Q | -- quiet Print less (only print events). # Print monitoring event information
-M | -- monitor Keep listening for events forever. Without this option, inotifywait will exit after one event is already ed. # always Keep the event listening status
-- Excludei <pattern> Like -- exclude but case insensitive. # case insensitive when excluding files or directories.
-- Timefmt <fmt> strftime-compatible format string for use with % T in -- format string. # specify the time output format
-- Format <fmt> Print using a specified printf-like format string; read the man page for more details.
# Print a string in a format similar to the specified output
-E | -- event <event1> [-e | -- event <event2>...] listen for specific event (s ). if omitted, all events are listened. # You can use this parameter to specify the event to be monitored, as shown below:
Events:
Access file or directory contents were read # the file or directory is read.
Modify file or directory contents were written # the file or directory content is modified.
Attrib file or directory attributes changed # the file or directory attribute is changed.
Close file or directory closed, regardless of read/write mode # the file or directory is closed, regardless of the read/write mode.
Open file or directory opened # the file or directory is opened.
Moved_to file or directory moved to watched directory # the file or directory is moved to another directory.
Move file or directory moved to or from watched directory # the file or directory is moved to another directory or from another directory to the current directory.
Create file or directory created within watched directory # the file or directory is created in the current directory.
Delete file or directory deleted within watched directory # the file or directory is deleted.
Unmount file system containing file or directory unmounted # the file system is uninstalled.

5.1.4 write a monitoring script and load it to the background for execution
[Root @ inotify-master scripts] # cat inotify. sh
#! /Bin/bash
# Para
Host01 = 192.168.1.160 # IP address of inotify-slave
Src =/backup/# local monitored directory
Dst = backup # Module name of the inotify-slave rsync Service
User = rsync_backup # inotify-slave rsync service virtual user
Rsync_passfile =/etc/rsync. password # the password file that calls the rsync Service locally
Inotify_home =/usr/local/inotify-3.14 # inotify installation directory
# Judge
If [! -E "$ src"] \
| [! -E "$ {rsync_passfile}"] \
| [! -E "$ {inotify_home}/bin/inotifywait"] \
| [! -E "/usr/bin/rsync"];
Then
Echo "Check File and Folder"
Exit 9
Fi
$ {Inotify_home}/bin/inotifywait-mrq -- timefmt '% d/% m/% y % H: % m' -- format' % T % w % F'-e close_write, delete, create, attrib $ src \
| While read file
Do
# Rsync-avzP -- delete -- timeout = 100 -- password-file =$ {rsync_passfile} $ src $ user @ $ host01: $ dst>/dev/null 2> & 1
Cd $ src & rsync-aruz-R -- delete. /-- timeout = 100 $ user @ $ host01: $ dst -- password-file =$ {rsync_passfile}>/dev/null 2> & 1
Done
Exit 0
[Root @ inotify-master scripts] # sh inotify. sh & # Add the script to the background for execution
[1] 13438
[Root @ inotify-master scripts] #

5.1.5 real-time synchronization test
 
Inotify-master operation:
[Root @ inotify-master scripts] # cd/backup/
[Root @ inotify-master backup] # ll
Total usage 0
[Root @ inotify-master backup] # for a in 'seq 100'; do touch $ a; done # create 200 files
[Root @ inotify-master backup] # ll -- time-style = full-iso
Total usage 0
-Rw-r -- 1 root 0 15:34:08. 141497569 + 0800 1
-Rw-r -- 1 root 0 15:34:08. 172497529 + 0800 10
-Rw-r -- 1 root 0 15:34:08. 235497529 + 0800 100
-Rw-r -- 1 root 0 15:34:08. 236497529 + 0800 101
-Rw-r -- 1 root 0 15:34:08. 237497529 + 0800 102
...................................

Inotify-slave check
[Root @ inotify-slave backup] # ll -- time-style = full-iso
Total usage 0
-Rw-r --. 1 rsync 0 15:34:08. 393823754 + 0800 1
-Rw-r --. 1 rsync 0 15:34:08. 393823754 + 0800 10
-Rw-r --. 1 rsync 0 15:34:08. 393823754 + 0800 100
-Rw-r --. 1 rsync 0 15:34:08. 393823754 + 0800 101
-Rw-r --. 1 rsync 0 15:34:08. 393823754 + 0800 102
..........................

RSync for file backup Synchronization

Monitor host files and directories using inotifywait

Using inotify + rsync for Linux File batch update

Inotify-tools + rsync real-time file synchronization installation and configuration

Complete rsync synchronization Configuration

Remote synchronization of Rsync in CentOS 6.5

Differential file comparison and extraction in Linux folders-the use of rsync

Rsync details: click here
Rsync: click here

This article permanently updates the link address: