Rsync+sersync+inotify to implement file synchronization between servers

Step One: Rsync installation configuration

Installing and using rsync

Installation of 2.1 rsync

Each mirror server (the server that needs to be synchronized) requires the Rsync daemon to be installed.

The general release comes with rsync, so I use the Yum installation directly here. Of course, on many distributions, Rsync has been installed by default.

The code is as follows

Copy Code

[Root@web1 ~]# Yum Install rsync [Root@web1 ~]# rpm-ql rsync /etc/xinetd.d/rsync /usr/bin/rsync /usr/share/doc/rsync-3.0.6 /usr/share/doc/rsync-3.0.6/copying /usr/share/doc/rsync-3.0.6/news /usr/share/doc/rsync-3.0.6/oldnews /usr/share/doc/rsync-3.0.6/readme /usr/share/doc/rsync-3.0.6/support /usr/share/doc/rsync-3.0.6/support/makefile /usr/share/doc/rsync-3.0.6/support/atomic-rsync /usr/share/doc/rsync-3.0.6/support/cvs2includes /usr/share/doc/rsync-3.0.6/support/deny-rsync /usr/share/doc/rsync-3.0.6/support/file-attr-restore /usr/share/doc/rsync-3.0.6/support/files-to-excludes /usr/share/doc/rsync-3.0.6/support/git-set-file-times /usr/share/doc/rsync-3.0.6/support/logfilter /usr/share/doc/rsync-3.0.6/support/lsh /usr/share/doc/rsync-3.0.6/support/mnt-excl /usr/share/doc/rsync-3.0.6/support/munge-symlinks /usr/share/doc/rsync-3.0.6/support/rrsync /usr/share/doc/rsync-3.0.6/support/rsyncstats /usr/share/doc/rsync-3.0.6/support/savetransfer.c /usr/share/doc/rsync-3.0.6/tech_report.tex /usr/share/man/man1/rsync.1.gz /usr/share/man/man5/rsyncd.conf.5.gz

Rsync is started by the Super Daemon xinetd. Therefore, the XINETD service needs to be started.

2.2 Configuring the Rsync server

The RSYNC server has three main profiles: Rsyncd.conf,rsyncd.secrets and RSYNCD.MOTD, and rsync does not automatically create the associated profile after installation, so we also need to manually create:

The code is as follows	Copy Code
[Root@web1 ~]# MKDIR/ETC/RSYNCD # #创建配置目录 [Root@web1 ~]# touch/etc/rsyncd/rsyncd.conf # #创建主配置文件 [Root@web1 ~]# touch/etc/rsyncd/rsyncd.secrets # #创建用户密码文件 [Root@web1 ~]# chmod 600/etc/rsyncd/rsyncd.secrets # #修改用户密码文件 [Root@web1 ~]# TOUCH/ETC/RSYNCD/RSYNCD.MOTD # #创建定义服务器信息的文件

To edit a master configuration file:

The code is as follows

Copy Code

[Root@web1 ~]# vi/etc/rsyncd/rsyncd.conf # Minimal configuration file for rsync daemon # to Rsync (1) and rsyncd.conf (5) man pages for help # This are required by THE/ETC/INIT.D/RSYNCD script PID file =/var/run/rsyncd.pid Port = 873 # #监听端口 address = 192.168.204.129 # #监听地址 #uid = Nobody #gid = Nobody UID = root GID = root Use chroot = yes # #是否限制在指定目录, in order to install, you typically need to enable Read Only = no #limit access to private LANs Hosts allow=192.168.204.0/255.255.255.0 # #允许网段 Hosts deny=* Max connections = 5 MOTD file =/ETC/RSYNCD/RSYNCD.MOTD #This would give you a separate log file #log file =/var/log/rsync.log #This'll log every file transferred-up to 85,000+ per user, per sync #transfer logging = yes Log format =%t%a%m%f%b Syslog facility = Local3 Timeout = 300 # #定义一个同步目录 [Webhome] Path =/var/www/html List=yes Ignore errors Auth users = Apache Secrets file =/etc/rsyncd/rsyncd.secrets # #指定上述账号密码文件 Comment = Web Home exclude = data/# #排除目录

Edit/etc/rsyncd/rsyncd.secrets File:

The code is as follows	Copy Code
[Root@web1 ~]# Vi/etc/rsyncd/rsyncd.secrets Apache:password123

# #注意, this account is the system account, but does not use the System account password, but the custom password

Restart to enable rsync to take effect:

The code is as follows	Copy Code
[Root@web1 ~]#/usr/bin/rsync--daemon--config=/etc/rsyncd/rsyncd.conf

Set to boot:

The code is as follows	Copy Code
[Root@web1 ~]# echo "/usr/bin/rsync--daemon--config=/etc/rsyncd/rsyncd.conf" >>/etc/rc.d/rc.local

2.3 Implementing data synchronization using Rsync clients

The client installs Ditto, the extra only need to configure a password file, specifically as follows:

The code is as follows	Copy Code
[Root@web2 ~]# MKDIR/ETC/RSYNCD # #创建配置目录 [Root@web2 ~]# echo "passowd123" >/etc/rsyncd/rsyncd.password # #创建密码文件, the password is the password in the server-side rsyncd.secrets file. [Root@web2 ~]# chmod 600/etc/rsyncd/rsyncd.password

Synchronous:

The code is as follows	Copy Code
[Root@web2 ~]# rsync-avzp--delete--password-file=/etc/rsyncd/rsyncd.password apache@web1::webhome/var/www/html/ Receiving incremental file list ./ Index.html 100% 21.48kb/s 0:00:00 (xfer#1, TO-CHECK=1/3) Newpage.html 100% 0.63kb/s 0:00:00 (xfer#2, TO-CHECK=0/3) Sent bytes received 264 bytes 728.00 Total size is speedup is 0.13

Description

--delete: Local and server exactly the same, if there is a different local, then delete, use caution

--password-file: Specifies the password file, if not specified, you need to manually enter

-A: parameter, equivalent to-rlptgod,-r is recursive-L is a linked file, meaning copy linked file,-p means to maintain the original permissions of the file,-T to maintain the original time of the file,-G to maintain the original user group,-O to maintain the original owner of the file;
-Z: Compressed when transmitting;
-P: Transmission progress;
-V: Information about progress during transmission

Apache@web1::webhome use a double colon to refer to a defined resource in the primary configuration file, or to use single quotes and absolute paths

When you synchronize later, only the new content is synchronized.

Finally, the above command is added to the scheduled task to synchronize.

Possible problems:

Firewall port Development issue: Iptables-a input-p tcp-m State--state new-m TCP--dport 873-j ACCEPT
Permission problem, write permission required

Step Two: Sersync+inotify to achieve synchronization

Before using Sersync, we must install the rsync server. What we need to be aware of here is that in the pure use of rsync for one-way synchronization, rsync daemon is run on the file push server, and the receiving server is running the rsync client. Using Sersync to do real-time synchronization of files just the opposite, the server that is used to receive files runs the rsync daemon.

First, install Sersync:

Download Address: Https://code.google.com/p/sersync/downloads/list

But note, due to our great Firewall, this address may not open, may need to turn over the wall to open.

Optional download package is free of compiler binary package and compile installation source code. (www.111cn.Net) I take binary as an example, the download contains a configuration file and an executable file.

The code is as follows

Copy Code

[Root@web1 gnu-linux-x86]# LL Total dosage 1772 -rwxr-xr-x 1 root 2214 October confxml.xml -rwxr-xr-x 1 root 1810128 October SERSYNC2 For ease of use, we copy it to the following directory: [Root@web1 gnu-linux-x86]# Mkdir/usr/local/sersync [Root@web1 gnu-linux-x86]# MV */usr/local/sersync Second, the basic configuration: 2.1 Creating a password file: [Root@web1 sersync]# pwd /usr/local/sersync [Root@web1 sersync]# echo "password@123" > RSYNC.PASSWD [Root@web1 sersync]# chmod rsync.passwd 2.2 Edit the main configuration file: The following red section is modified for me: [Root@web1 sersync]# VI Confxml.xml <?xml version= "1.0" encoding= "Iso-8859-1"?> <debug start= "false"/> <filesystem xfs= "false"/> < filter specific files, and files for specific folders during!--monitoring events--> <filter start= "true" > <exclude expression= "(. *). html" ></exclude> <exclude expression= "(. *). php" ></exclude> <exclude expression= "(. *). jpg" ></exclude> <exclude expression= "(. *). gif" ></exclude> <exclude expression= "(. *). png" ></exclude> <exclude expression= "(. *). css" ></exclude> <exclude expression= "(. *). js" ></exclude> <exclude expression= "^tmp/*" ></exclude> </filter> <!--set the events to monitor--> <inotify> <delete start= "true"/> <createfolder start= "true"/> <createfile start= "true"/> <closewrite start= "true"/> <movefrom start= "true"/> <moveto start= "true"/> <attrib start= "false"/> <modify start= "true"/> </inotify> <sersync> <!--set up the monitoring directory--> <localpath watch= "/var/www/html" > <!--specifies the address and module name of the remote rsync--> <remote ip= "192.168.204.129" name= "Webhome"/> <!--<remote ip= "192.168.204.128" name= "Tongbu"/>--> <!--<remote ip= "192.168.8.40" name= "Tongbu"/>--> </localpath> <rsync> <commonparams params= "-artuz"/> <!--enable authentication and specify a password to store the file--> <auth start= "true" users= "Apache" passwordfile= "/usr/local/sersync/rsync.passwd"/> <userdefinedport start= "false" port= "874"/><!--port=874--> <timeout start= "false" Time= "/><!--timeout=100--> <ssh start= "false"/> </rsync> <faillog path= "/tmp/rsync_fail_log.sh" timetoexecute= "/><"!--default every 60mins execute once-> <!--whether to enable full rsync execution and specify the execution cycle--> <crontab start= "true" schedule= "1200" ><!--600mins--> <crontabfilter start= "true" > <exclude expression= "(. *). html" ></exclude> <exclude expression= "(. *). php" ></exclude> <exclude expression= "(. *). jpg" ></exclude> <exclude expression= "(. *). gif" ></exclude> <exclude expression= "(. *). png" ></exclude> <exclude expression= "(. *). css" ></exclude> <exclude expression= "(. *). js" ></exclude> <exclude expression= "^tmp/*" ></exclude> </crontabfilter> </crontab> <plugin start= "false" name= "command"/> </sersync> &lt;plugin name= "command" &gt; &lt;param prefix= "/bin/sh" suffix= "ignoreerror=" true "/&gt; &lt;!--prefix/opt/tongbu/mmm.sh suffix--&gt; &lt;filter start= "false" &gt; &lt;include expression= "(. *). php"/&gt; &lt;include expression= "(. *). Sh"/&gt; &lt;/filter&gt; &lt;/plugin&gt; &lt;plugin name= "Socket" &gt; &lt;localpath watch= "/opt/tongbu" &gt; &lt;deshost ip= "192.168.138.20" port= "8009"/&gt; &lt;/localpath&gt; &lt;/plugin&gt; &lt;plugin name= "Refreshcdn" &gt; &lt;localpath watch= "/data0/htdocs/cms.xoyo.com/site/" &gt; &lt;cdninfo domainname= "ccms.chinacache.com" port= "xxx" username= "xxxx" passwd= "xxxx"/&gt; &lt;sendurl base= "Http://pic.xoyo.com/cms"/&gt; &lt;regexurl regex= "false" Match= cms.xoyo.com/site ([/a-za-z0-9]*). Xoyo.com/images "/&gt;" &lt;/localpath&gt; &lt;/plugin&gt; &lt;/head&gt;

Third, enable the synchronization function:

Again, all Sersync clients need to install rsync and start as a service. All client IP (that is, rsync server side) that needs to be synchronized must be joined in the above configuration file.

Initialize the synchronization using the following command and monitor the specified directory changes:

The code is as follows

Copy Code

[Root@web2 sersync2]#./sersync2-r Set the system param Execute:echo 50000000 >/proc/sys/fs/inotify/max_user_watches Execute:echo 327679 >/proc/sys/fs/inotify/max_queued_events Parse the command param Option:-R rsync All of the local files to the remote servers before the Sersync work Daemon Thread Num:10 Parse XML config file Host Ip:localhost host port:8008 Use rsync password-file: User is Apache Passwordfile Is/etc/rsyncd/rsyncd.password Config XML parse success Please set/etc/rsyncd.conf Max connections=0 manually Sersync Working thread = 1 (primary thread) + 1 (fail retry thread) + (daemon sub threads) Max Threads Numbers is:22 = (Thread pool nums) + (Sub threads) Please according your CPUs, use-n param to adjust the CPU rate ------------------------------------------ Rsync the directory recursivly to the remote servers once Working Please wait ... Execute command:cd/var/www/html && rsync-artuz-r--delete./--port=873 Apache@192.168.204.129::webhome--pas Sword-file=/etc/rsyncd/rsyncd.password >/dev/null 2>&1 Run the Sersync: Watch path is:/var/www/html

You can specify the number of threads to execute concurrently with the following parameters:

The code is as follows	Copy Code
./sersync-n Num

Other parameters:

Parameter-D: Daemon mode enabled
Parameter-R: Push the Monitor directory with the remote host with the rsync command before monitoring
C parameter-N: Specifies the number of daemon threads to open, default to 10
Parameter-o: Specifies the configuration file, using the Confxml.xml file by default
Parameter-M: Enable additional modules separately, use-M refreshcdn to open the Refresh CDN module
Parameter-M: enable other modules separately, using the-m socket to open the socket module
Parameter-M: enable other modules separately, use-m HTTP to open the HTTP module
Synchronization is performed by default without the-m parameter
Join Boot:

code is as follows	copy code
[ROOT@WEB2 sersync2]# echo "/usr/local/sersync2/sersync2-d-o/usr/local/sersync2/confxml.xml" >>/etc/rc.d/rc.local