Rsync+inotify for real-time synchronization

1. Rsync+inotify Synchronization Logic diagram

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M02/24/E7/wKiom1NWH9zSygnhAAESOQCBlMs996.jpg "alt=" Wkiom1nwh9zsygnhaaesoqcblms996.jpg "/>

2. Environment deployment

Host Name	Host IP Address	System version	System Kernel Version
Inotify-master	192.168.30.130	CentOS 6.5	2.6.32-358.el6.x86_64
Inotify-slave	192.168.30.131	CentOS 6.5	2.6.32-358.el6.x86_64

3.1 Inotify-slave Deployment

Here is the deployment of rsync service, rsync daemon working mode.

3.1.1 Check if rsync is installed

[[email protected] ~]# rpm-aq rsyncrsync-3.0.6-9.el6.x86_64[[email protected] ~]# yum-y install rsync xinetd #如果没有安装, so Installing rsync with Yum

3.1.2 Create a new rsync user and Module directory and change its user group

[Email protected] mail]# useradd rsync-s/sbin/nologin-m #添加rsync用户 [[email protected] mail]# grep rsync/etc/passwdrsy Nc:x:2004:2004::/home/rsync:/sbin/nologin[[email protected] mail]# mkdir/backup #创建rsync daemon working mode module directory [email Protected] mail]# ll-d/backup/drwxr-xr-x.  2 root root 4096 April 14:13/backup/[[email protected] mail]# chown rsync.rsync/backup/#更改模块目录的用户组 [[email protected] mail]# ll-d/backup/drwxr-xr-x. 2 rsync rsync 4096 April 14:13/backup/

3.1.3 Writing the rsync daemon configuration file/etc/rsyncd.conf

[Email protected]/]# cat/etc/rsyncd.conf# #rsyncd. conf start## #工作中指定用户 (requires user specified) uid = Rsyncgid = rsync# equals black hole. Error locating use Chroot = no# How many clients are simultaneously transmitting files max connections = 200# time-out timeout = 300# process Number file pid file =/var/run/rsyncd.pid# log Files "lock file =/var/r" un/rsync.lock# log file =/var/log/rsyncd.log# Module Start # This module corresponds to the push directory # module name random [backup] #需要同步的目录path =/backup/# Indicates an error was ignored error ignore errors# indicates network permissions writable (local control is really writable) Read Only = false# here set IP or let sync list = false# Specify allowed network segment hosts allow = 192.168.30.0/ 24# rejects the link's address, indicating that there are no rejections. Hosts Deny = 0.0.0.0/32# things (by default) #虚拟用户auth users = rsync_backup# password files for virtual users Secrets file =/etc/rsync.password# End of profile # Rsync_config_______________end

3.1.4 Configuring a virtual user's password file

[Email protected]/]# echo "Rsync_backup:leesir" >/etc/rsync.password[[email protected]/]# cat/etc/ Rsync.passwordrsync_backup:leesir #注: Rsync_backup for Virtual user, Leesir for this virtual user's password [[email protected]/]# chmod 600/etc/ Rsync.password #为密码文件提权, added security [[email protected]/]# LL/ETC/RSYNC.PASSWORD-RW-------. 1 root root 20 April 14:20/etc/rsync.password

3.1.5 Start rsync Service

[[email protected] /]# rsync --daemon    #启动rsync服务 [[email protected]  /]# ps -ef |grep rsyncroot     14871      1  0 14:24 ?        00:00:00  rsync --daemonroot     14873 14788  0 14:24 pts/0     00:00:00 grep rsync[[email protected] /]# netstat -lnutp  |grep rsynctcp        0      0  0.0.0.0:873                  0.0.0.0:*                    LISTEN      14871/rsynctcp         0      0 :::873                       :::*                           Listen      14871/rsync

3.1.6 via Inotify-master test push

Inotify-master Configure password file, test push

[[email protected] ~]# echo  "Leesir"  >/etc/ rsync.password[[email protected] ~]# cat /etc/rsync.passwordleesir   # Note: Just write the password here, remember. [[email protected] ~]# chmod 600 /etc/rsync.password[[email protected] ~]#  LL /ETC/RSYNC.PASSWORD-RW------- 1 root root 7 4 Month   22 14:32  /etc/rsync.password[[email protected] ~]# echo  "Hello leesir" >test.txt[[ email protected] ~]# cat test.txthello leesir[[email protected] ~]#  rsync -avz test.txt [email protected]::backup --password-file=/etc/ Rsync.passwordsending incremental file listtest.txtsent 82 bytes  received  27 bytes  72.67 bytes/sectotal size is 13  speedup is  0.12 

Inotify-slave Check:

[[email protected]/]# ll/backup/total usage 4-rw-r--r--. 1 rsync rsync 13 April 14:34 test.txt[[email protected]/]# Cat/backup/test.txthello Leesir

4.1 Inotify-master Deployment

Note:

INotify is installed and executed by the Rsync client

Enterprise Scene Stress test 200-300 synchronization limit, by the network card, disk, bandwidth and other constraints.

4.1.1 See if the current system supports inotify

[[email protected] ~]# ll/proc/sys/fs/inotify/Total usage 0-rw-r--r--1 root root 0 April 14:56 max_queued_events-rw-r--r--1 ro OT root 0 April 14:56 max_user_instances-rw-r--r--1 root root 0 April 14:56 max_user_watches# show that three files prove support.

Expand:

/proc/sys/fs/inotify/max_queued_evnets

Represents the maximum value assigned to the number of events that can be queued in inotify instance when calling Inotify_init, an event that exceeds this value is discarded, but the In_q_overflow event is triggered.

/proc/sys/fs/inotify/max_user_instances

Represents the maximum number of inotify instatnces that each real user ID can create.

/proc/sys/fs/inotify/max_user_watches

Represents the maximum number of directories that can be monitored per inotify instatnces. If you monitor a large number of files, you need to increase the size of this value appropriately, depending on the situation.

Example: Echo 30000000 >/proc/sys/fs/inotify/max_user_watches

4.1.2 Download INotify source package and compile the installation

[Email protected] tools]# wget http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz # Download the INotify source package ......... ................ [email protected] tools]# ll inotify-tools-3.14.tar.gz-rw-r--r--1 root root 358772 March inotify-tools-3.14.tar.gz[ [Email protected] tools]# tar zxf inotify-tools-3.14.tar.gz[[email protected] tools]# CD Inotify-tools-3.14[[email Protected] inotify-tools-3.14]#./configure--prefix=/usr/local/inotify-3.14 #配置inotify, and specify the installation path to/usr/local/ Inotify-3.14 ......... ............... [[email protected] inotify-tools-3.14]# make && make install

4.1.3 inotify inotifywait Command Common parameters detailed

[[email protected] inotify-tools-3.14]# cd /usr/local/inotify-3.14/[[email protected]  inotify-3.14]# ./bin/inotifywait --help-r|--recursive   watch directories  recursively.  #递归查询目录-q|--quiet      print less  (only  print events) .  #打印监控事件的信息-m|--monitor   keep listening for events  forever.  Without this option, inotifywait will exit after  one  event is received.        # Always keep the event listening state--excludei <pattern>  like --exclude but case insensitive.      #排除文件或目录时, case insensitive. --timefmt <fmt> strftime-compatible format string for use with % t in --format string.  #指定时间输出的格式--format <fmt>  print using a specified printf-like format string; read the man page  For more details. #打印使用指定的输出类似格式字符串-e|--event <event1> [ -e|--event < Event2> ... ] listen for specific event (s) .  if omitted,  all events are  listened for.    #通过此参数可以指定需要监控的事件 as follows: events:access            file or directory contents  were read        #文件或目录被读取. modify           file or directory  contents were written     #文件或目录内容被修改. Attrib            file or directory  attributes changed       #文件或目录属性被改变. close             file or directory closed, regardless of  read/write mode     #文件或目录封闭 Regardless of read/write mode. open            file or directory  opened                      #文件或目录被打开. moved_to        file or directory moved to  watched directory     #文件或目录被移动至另外一个目录. move            file or directory  moved to or from watched directory    # The file or directory is moved to another directory or moved from another directory to the current directory. create           file or directory  created within watched directory      #文件或目录被Create in current directory Delete           file or directory  deleted within watched directory      #文件或目录被删除unmount           file system containing file or  directory unmounted   #文件系统被卸载

4.1.4 Write a monitoring script and load it into the background execution

[[email protected] scripts]# cat inotify.sh#!/bin/bash#parahost01=192.168.30.130    #inotify-slave IP address src=/backup/         #本地监控的目录dst =backup           module Name of the Rsync service for the #inotify-slave user=rsync_backup        Virtual User rsync_passfile=/etc/rsync.password   # #inotify-slave's rsync service Local call to rsync service password file inotify_home=/usr/local/inotify-3.14     #inotify的安装目录 #judgeif [  ! -e  "$SRC"  ] | |  [ ! -e  "${rsync_passfile}"  ] | |  [ ! -e  "${inotify_home}/bin/inotifywait"  ] | |  [ ! -e  "/usr/bin/rsync"  ]thenecho  "Check file and folder" exit  9fi${inotify_home}/bin/inotifywait -mrq --timefmt  '%d/%m/%y %h:%m '  --format   '%t %w%f '  -e close_write,delete,create,attrib  $src   | while read filedo#  rsync -avzp --delete --timeout=100 -- password-file=${rsync_passfile}  $src  [email protected] $host:: $dst  >/dev/null 2 >&1cd  $src  && rsync -aruz -r --delete ./  -- Timeout=100 [email protected] $host:: $dst  --password-file=${rsync_passfile} >/dev/null  2>&1doneexit 0[[email protected] scripts]# sh inotify.sh &    #将脚本加入后台执行 [1] 13438[[email protected] scripts]#

4.1.5 Real-time synchronization test

Inotify-master Operation:

[[email protected] scripts]# Cd/backup/[[email protected] backup]# ll total dosage 0[[email protected] backup]# for a in ' SEQ 200 '; Do touch $a;d one #创建200个文件 [[email protected] backup]# ll--time-style=full-iso total usage 0-rw-r--r--1 root root 0 2014-04-22 1 5:34:08.141497569 +0800 1-rw-r--r--1 root root 0 2014-04-22 15:34:08.172497529 +0800 10-rw-r--r--1 root root 0 2014-04-2 2 15:34:08.235497529 +0800 100-rw-r--r--1 root root 0 2014-04-22 15:34:08.236497529 +0800 101-rw-r--r--1 root root 0 201 4-04-22 15:34:08.237497529 +0800 102

Inotify-slave Check

[email protected] backup]# ll--time-style=full-iso total dosage 0-rw-r--r--. 1 rsync rsync 0 2014-04-22 15:34:08.393823754 +0800 1-rw-r--r--. 1 rsync rsync 0 2014-04-22 15:34:08.393823754 +0800 10-rw-r--r--. 1 rsync rsync 0 2014-04-22 15:34:08.393823754 +0800 100-rw-r--r--. 1 rsync rsync 0 2014-04-22 15:34:08.393823754 +0800 101-rw-r--r--. 1 rsync rsync 0 2014-04-22 15:34:08.393823754 +0800 102

This article is from the "Healer" blog, make sure to keep this source http://zlyang.blog.51cto.com/1196234/1687706

Rsync+inotify for real-time synchronization