Rsync+inotify enables real-time backup of data

I. Overview of Rsync 1.1 , rsync the advantages and disadvantages

Rsync, compared with traditional CP and tar backup methods, has the advantages of high security, fast backup, support of incremental backup, and so on, rsync can solve the requirement of low-real-time data backup, such as regular backup file server data to remote server, regular data mirroring for local disk, etc.
With the expansion of application system scale, the security and reliability of data also put forward better requirements, rsync in the high-end business system also gradually exposed a lot of shortcomings, first, rsync synchronization data, need to scan all files after the comparison, for differential transmission. If the number of files reaches millions or even tens of thousands of levels, scanning all the files will be very time consuming. And what is changing is often a small part of it, which is a very inefficient way. Second, rsync cannot monitor and synchronize data in real time, although it can trigger synchronization through the way of the Linux daemon, but there must be a time difference between the two triggering actions, which results in inconsistent service and client data and the inability to fully recover data in the event of an application failure. Based on the above reasons, the rsync+inotify combination appeared!

Ii. Overview of INotify 2.1 , Primary knowledge inotify

INotify is a powerful, fine-grained, asynchronous file system event monitoring mechanism, the Linux kernel from 2.6.13, joined the INotify support, through INotify can monitor the file system to add, delete, modify, move and other subtle events, using this kernel interface, Third-party software can monitor the file system under the various changes, and inotify-tools is such a third-party software.
In the above section, we mentioned that rsync can achieve triggered file synchronization, but through the crontab daemon mode trigger, synchronous data and actual data will be different, and inotify can monitor the file system changes, when the file has any changes, trigger rsync synchronization, This just solves the real-time problem of synchronizing data.

2.2 , inotify installation, in WEB Release the Management Server installation, WEB node does not need to be installed

tar–xvf inotify-tools-3.14. Tar . GZCD inotify-tools-3.14. /Configuremake doinstall/usr/local/bin/inotifywa*

-rwxr-xr-x 1 root root 44319 April 6 18:51/usr/local/bin/inotifywait

-rwxr-xr-x 1 root root 41409 April 6 18:51/usr/local/bin/inotifywatch

Inotifywait is used to wait for a specific event on a file or set of files, it can monitor any file and directory settings, and can recursively monitor the entire directory tree

Inotifywatch is used to collect monitored file system statistics, including the number of secondary messages per INotify event

III. Enterprise Case Web content release Management 3.1 , environmental description

As the business grows, the Web program maintains consistency across all web tiers. Distributed through a content server and radiate all of the updated web programs, the simulated environment is as follows:

WEB1 node ip:192.168.1.10

WEB2 node Content Publishing Server ip:192.168.1.5

3.2 , installation WEB1 node

Yum install-y rsync echo gongda:123 >/etc/web1.pass
chmod 600/etc/web1.pass
Mkdir/web1/wwwroot-p
Vim/etc/rsyncd.conf #此配置文件没有. Need to re-edit a

UID =Nobodygid=Nobodyusechroot=Nomax Connections=TenStrict Modes=Yespidfile=/var/run/Rsyncd.pidlockfile=/var/run/Rsync.locklogfile=/var/log/Rsyncd.log[web1]path=/web1/wwwroot/Comment= Web1fileIgnore Errorsread only=NoWriteonly =nohosts Allow=10.0.0.201hosts Deny= *List=falseUID=Rootgid=Rootauth Users=web1usersecretsfile=/etc/web1.pass

Rsyncd.conf

Start the Rsync service

# rsync--daemon

3.2 , installation WEB2 Content Management Node

Yum Install rsync

Sync the password, note here only to write the password, and Web1.pass the passwords inside are so

Echo 123 >/etc/server.pass
chmod 600/etc/server.pass

Write a startup script

Vim/opt/inotify-rsync. SH

#!/bin/Bashhost1=192.168.1.5#======================src=/web/wwwroot/Dst1=Web1user1=Daemon/usr/local/bin/inotifywait-mrq--timefmt'%d/%m/%y%h:%m'--format'%T%w%f%e'-e modify,delete,create,attrib $src| whileRead Files Do/USR/BIN/RSYNC-VZRTOPG--delete--progress--password-file=/etc/server.pass $src [email protected] $host 1:: $DST 1Echo "${files} was rsynced">>/tmp/rsync.log2>&1 Done

Startup scripts

CD/optchmod +x inotify-rsync. shnohup bash-x inotify-rsync.sh &

Inotifywait

Grammar:
inotifywait [-HCMRQ] [-e] [-t] [--format][--TIMEFMT] [...]
Parameters:

-h,–help output Help information @ excludes files that you do not need to monitor, either as a relative or an absolute path. –fromfile read from a file to be monitored or excluded files, a file line, excluded files at @ start. -M,–monitor receives a thing without exiting and executes indefinitely. The default behavior is to exit immediately after receiving a thing. -D,–daemon and –monitor, in addition to running in the background, you need to specify –outfile to export things to a file. Also means the use of –syslog. -o,–outfile output Things to a file instead of the standard output. -s,–syslog output error message to System log-r,–recursive monitors all subdirectories in a directory. -Q,–quiet specified once, does not output details, specified two times, except for fatal errors, will not output any information. –exclude Regular matching files that need to be excluded are case-sensitive. –excludei matches files that need to be excluded, ignoring case. -T, –timeout sets the timeout period, and if 0, executes indefinitely. -E, –event specifies the monitored events. -c,–csv output CSV format. –TIMEFMT specifies the time format for the –format option in the%t format. –FORMAT specifies the output format. %Windicates the directory where the event occurred%f indicates the file where the event occurred%e Indicates the event that occurred%Xe events are separated by "X"%T using the time format defined by –TIMEFMT

inotifywait Parameter IntroductionInotifywatch

Grammar:
Inotifywatch [-HVZRQF] [-e] [-t] [-a] [-d] [...]

-h,–help Output Help information -v,–verbose Output details @ Exclude files that do not need to be monitored, either relative or absolute. –fromfile read from a file to be monitored or excluded files, a file line, excluded files at @ start. -Z,–zero The rows and columns of the output table, even if the element is empty –exclude regular matches the file that needs to be excluded, case sensitive. –excludei matches files that need to be excluded, ignoring case. -r,–recursive monitors all subdirectories in a directory. -T, –timeout set timeout -E, –event only listens for the specified event. -A, –ascending in ascending order of specified events. -D, –descending in descending order of specified events. 

Inotifywatch Parameter Introduction

3.2 , common errors

Question one:

@ERROR: Chroot failed

Rsync Error:error starting Client-server Protocol (code 5) at MAIN.C (1522) [receiver=3.0.3]

Cause: The server-side directory does not exist or has no permissions. Creating a directory and correcting permissions can resolve the issue.

Question two:

@ERROR: Auth failed on module tee

Rsync Error:error starting Client-server Protocol (code 5) at MAIN.C (1522) [receiver=3.0.3]

Cause: The server side of the module (tee) needs to verify the user name password, but the client does not provide the correct user name password, authentication failed. Provide the correct user name password to resolve this issue.

Question three:

@ERROR: Unknown module ' tee_nonexists '

Rsync Error:error starting Client-server Protocol (code 5) at MAIN.C (1522) [receiver=3.0.3]

Cause: The server does not exist for the specified module. Provide the correct module name or modify the server side to the module you want to solve the problem.

Question four:

Password file must not being other-accessible

Continuing without password file

Password:

Cause: This is because the permissions of the Rsyncd.pwd rsyncd.secrets are not correct and should be set to 600. such as: chmod rsyncd.pwd

Question five:

rsync:failed to connect to 218.107.243.2:no route to host (113)

Rsync error:error in Socket IO (code ten) at CLIENTSERVER.C (104) [receiver=2.6.9]

Cause: The other side does not boot, firewall blocking, through the network has a firewall blocking, it is possible. Shutting down the firewall is actually opening the TCPUDP 873 port.

Question six:

Rsync Error:error starting Client-server Protocol (code 5) at MAIN.C (1524) [receiver=3.0.7]

Cause: There was an error in the contents of the/etc/rsyncd.conf configuration file. Please check the configuration file correctly.

Question seven:

Rsync:chown "" failed:invalid argument (22)

Cause: Permissions cannot be copied. Remove the parameters for the sync permission. (This is often the case when Linux is to Windows)

Question eight:

@ERROR: Daemon Security Issue--contact admin
Rsync Error:error starting Client-server Protocol (code 5) at MAIN.C (1530) [sender=3.0.6]

Cause: There is a soft connection file in the directory of the synchronization, need the server side/etc/rsyncd.conf Open Use chroot = yes. Skim the soft connection file.

Rsync+inotify enables real-time backup of data