2015-07-07 rsync+inotify for real-time synchronization

Transferred from: http://chocolee.blog.51cto.com/8158455/1400596

With the expansion of application system scale, the security and reliability of data also put forward better requirements, rsync in the high-end business system also gradually exposed a lot of shortcomings, first, rsync synchronization data, need to scan all files after the comparison, for differential transmission. If the number of files reaches millions or even tens of thousands of levels, scanning all the files will be very time consuming. And what is changing is often a small part of it, which is a very inefficient way. Second, rsync cannot monitor and synchronize data in real time, although it can trigger synchronization through the way of the Linux daemon, but there must be a time difference between the two triggering actions, which results in inconsistent service and client data and the inability to fully recover data in the event of an application failure. Based on the above reasons, the rsync+inotify combination appeared!

1.1 INotify Introduction

2.1 Rsync+inotify Synchronization Logic diagram

3.1 Environment Deployment

4.1 Inotify-slave Deployment

4.1.1 Check if rsync is installed

4.1.2 Create a new rsync user and Module directory and change its user group

4.1.3 Writing the rsync daemon configuration file/etc/rsyncd.conf

4.1.4 Configuring a virtual user's password file

4.1.5 Start rsync Service

4.1.6 via Inotify-master test push

5.1 Inotify-master Deployment

5.1.1 See if the current system supports inotify

5.1.2 Download INotify source package and compile the installation

5.1.3 inotify inotifywait Command Common parameters detailed

5.1.4 Write a monitoring script and load it into the background execution

5.1.5 Real-time synchronization test

1.1 INotify Introduction

INotify is a powerful, fine-grained, asynchronous file system event control mechanism. Linux kernel from 2.6.13, joined the INotify support, through the inotify can monitor the file system to add, delete, modify, move and other events, using this kernel interface, third-party software can monitor the file system under the various changes in the file, And Inotify-tools is the implementation of monitoring software.

2.1 Rsync+inotify Synchronization Logic diagram

3.1 Environment Deployment

Host Name

Host IP Address

System version

System Kernel Version

Inotify-master

192.168.1.128

CentOS Release 6.4 (Final)

2.6.32-358.el6.x86_64

Inotify-slave

192.168.1.160

CentOS Release 6.4 (Final)

2.6.32-358.el6.x86_64

4.1 Inotify-slave Deployment

Here is the deployment of rsync service, rsync daemon working mode.

4.1.1 Check if rsync is installed

[[email protected] ~]# rpm -aq rsync

rsync-3.0.6-9.el6.x86_64

4.1.2 Create a new rsync user and Module directory and change its user group

[[email protected] mail]# useradd rsync -s /sbin/nologin  -M #添加rsync用户

[[email protected] mail]# grep rsync /etc/passwd

rsync:x:2004:2004::/home/rsync:/sbin/nologin

[[email protected] mail]# mkdir /backup  #创建rsync daemon工作模式的模块目录

[[email protected] mail]# ll -d /backup/

drwxr-xr-x. 2 root root 4096 4月  22 14:13 /backup/

[[email protected] mail]# chown rsync.rsync /backup/   #更改模块目录的用户组

[[email protected] mail]# ll -d /backup/

drwxr-xr-x. 2 rsync rsync 4096 4月  22 14:13 /backup/

4.1.3 Writing the rsync daemon configuration file/etc/rsyncd.conf

[[email protected] /]# cat /etc/rsyncd.conf

##rsyncd.conf start##

#工作中指定用户(需要指定用户)

uid = rsync

gid = rsync

#相当于黑洞.出错定位

use chroot = no

#有多少个客户端同时传文件

max connections = 200

#超时时间

timeout = 300

#进程号文件

pid file = /var/run/rsyncd.pid

#日志文件

lock file = /var/run/rsync.lock

#日志文件

log file = /var/log/rsyncd.log

#模块开始

#这个模块对应的是推送目录

#模块名称随便起

[backup]

#需要同步的目录

path = /backup/

#表示出现错误忽略错误

ignore errors

#表示网络权限可写(本地控制真正可写)

read only = false

#这里设置IP或让不让同步

list = false

#指定允许的网段

hosts allow = 192.168.1.0/24

#拒绝链接的地址，一下表示没有拒绝的链接。

hosts deny = 0.0.0.0/32

#不要动的东西(默认情况)

#虚拟用户

auth users = rsync_backup

#虚拟用户的密码文件

secrets file = /etc/rsync.password

#配置文件的结尾

#rsync_config_______________end

4.1.4 Configuring a virtual user's password file

[[email protected] /]# echo "rsync_backup:leesir" >/etc/rsync.password

[[email protected] /]# cat /etc/rsync.password

rsync_backup:leesir   #注：rsync_backup为虚拟用户，leesir为这个虚拟用户的密码

[[email protected] /]# chmod 600 /etc/rsync.password #为密码文件提权，增加安全性

[[email protected] /]# ll /etc/rsync.password

-rw-------. 1 root root 20 4月  22 14:20 /etc/rsync.password

4.1.5 Start rsync Service

[[email protected] /]# rsync --daemon   #启动rsync服务

[[email protected] /]# ps -ef |grep rsync

root     14871     1  0 14:24 ?        00:00:00 rsync --daemon

root     14873 14788  0 14:24 pts/0    00:00:00 grep rsync

[[email protected] /]# netstat -lnutp |grep rsync

tcp        0      0 0.0.0.0:873           0.0.0.0:*                   LISTEN      14871/rsync

tcp        0      0 :::873                :::*                        LISTEN      14871/rsync

4.1.6 via Inotify-master test push

Inotify-master Configure password file, test push

[[email protected] ~]# echo "leesir" >/etc/rsync.password

[[email protected] ~]# cat /etc/rsync.password

leesir   #注意：这里只要写密码即可，切记。

[[email protected] ~]# chmod 600 /etc/rsync.password

[[email protected] ~]# ll /etc/rsync.password

-rw------- 1 root root 7 4月  22 14:32 /etc/rsync.password

[[email protected] ~]# echo "hello leesir">test.txt

[[email protected] ~]# cat test.txt

hello leesir

[[email protected] ~]# rsync -avz test.txt [email protected]::backup --password-file=/etc/rsync.password

sending incremental file list

test.txt

sent 82 bytes  received 27 bytes  72.67 bytes/sec

total size is 13  speedup is 0.12

Inotify-slave Check:

[[email protected] /]# ll /backup/

总用量 4

-rw-r--r--. 1 rsync rsync 13 4月  22 14:34 test.txt

[[email protected] /]# cat /backup/test.txt

hello leesir

5.1 Inotify-master Deployment

Note:

INotify is installed and executed by the Rsync client

Enterprise Scene Stress test 200-300 synchronization limit, by the network card, disk, bandwidth and other constraints.

5.1.1 See if the current system supports inotify

[[email protected] ~]# ll /proc/sys/fs/inotify/

总用量 0

-rw-r--r-- 1 root root 0 4月  22 14:56 max_queued_events

-rw-r--r-- 1 root root 0 4月  22 14:56 max_user_instances

-rw-r--r-- 1 root root 0 4月  22 14:56 max_user_watches

#显示这三个文件则证明支持。

Expand:

/proc/sys/fs/inotify/max_queued_evnets

Represents the maximum value assigned to the number of events that can be queued in inotify instance when calling Inotify_init, an event that exceeds this value is discarded, but the In_q_overflow event is triggered.

/proc/sys/fs/inotify/max_user_instances

Represents the maximum number of inotify instatnces that each real user ID can create.

/proc/sys/fs/inotify/max_user_watches

Represents the maximum number of directories that can be monitored per inotify instatnces. If you monitor a large number of files, you need to increase the size of this value appropriately, depending on the situation.

Example: Echo 30000000 >/proc/sys/fs/inotify/max_user_watches

5.1.2 Download INotify source package and compile the installation

[email protected] tools]# wget http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz  #下载inotify源码包

..................................

[email protected] tools]# ll inotify-tools-3.14.tar.gz

-rw-r--r-- 1 root root 358772 3月  14 2010 inotify-tools-3.14.tar.gz

[[email protected] tools]# tar zxf inotify-tools-3.14.tar.gz

[[email protected] tools]# cd inotify-tools-3.14

[[email protected] inotify-tools-3.14]# ./configure --prefix=/usr/local/inotify-3.14 #配置inotify,并指定安装路径为/usr/local/inotify-3.14

................................

[[email protected] inotify-tools-3.14]# make && make install

................................

5.1.3 inotify inotifywait Command Common parameters detailed

[[email protected] inotify-tools-3.14]# cd /usr/local/inotify-3.14/

[[email protected] inotify-3.14]# ./bin/inotifywait --help

-r|--recursive   Watch directories recursively. #递归查询目录

-q|--quiet      Print less (only print events). #打印监控事件的信息

-m|--monitor   Keep listening for events forever.  Without this option, inotifywait will exit after one  event is received. #始终保持事件监听状态

--excludei <pattern>  Like --exclude but case insensitive.    #排除文件或目录时，不区分大小写。

--timefmt <fmt> strftime-compatible format string for use with %T in --format string. #指定时间输出的格式

--format <fmt>  Print using a specified printf-like format string; read the man page for more details.

#打印使用指定的输出类似格式字符串

-e|--event <event1> [ -e|--event <event2> ... ] Listen for specific event(s).  If omitted, all events are  listened for.   #通过此参数可以指定需要监控的事件，如下所示:

Events：

access           file or directory contents were read       #文件或目录被读取。

modify           file or directory contents were written    #文件或目录内容被修改。

attrib            file or directory attributes changed      #文件或目录属性被改变。

close            file or directory closed, regardless of read/write mode    #文件或目录封闭，无论读/写模式。

open            file or directory opened  #文件或目录被打开。

moved_to        file or directory moved to watched directory    #文件或目录被移动至另外一个目录。

move            file or directory moved to or from watched directory    #文件或目录被移动另一个目录或从另一个目录移动至当前目录。

create           file or directory created within watched directory     #文件或目录被创建在当前目录

delete           file or directory deleted within watched directory     #文件或目录被删除

unmount         file system containing file or directory unmounted  #文件系统被卸载

5.1.4 Write a monitoring script and load it into the background execution

[[email protected] scripts]# cat inotify.sh

#!/bin/bash

#para

host01=192.168.1.160  #inotify-slave的ip地址

src=/backup/        #本地监控的目录

dst=backup         #inotify-slave的rsync服务的模块名

user=rsync_backup      #inotify-slave的rsync服务的虚拟用户

rsync_passfile=/etc/rsync.password   #本地调用rsync服务的密码文件

inotify_home=/usr/local/inotify-3.14    #inotify的安装目录

#judge

if [ ! -e "$src" ] \

|| [ ! -e "${rsync_passfile}" ] \

|| [ ! -e "${inotify_home}/bin/inotifywait" ] \

|| [ ! -e "/usr/bin/rsync" ];

then

echo "Check File and Folder"

exit 9

fi

${inotify_home}/bin/inotifywait -mrq --timefmt ‘%d/%m/%y %H:%M‘ --format ‘%T %w%f‘ -e close_write,delete,create,attrib $src \

| while read file

do

#  rsync -avzP --delete --timeout=100 --password-file=${rsync_passfile} $src [email protected]$host01::$dst >/dev/null 2>&1

cd $src && rsync -aruz -R --delete ./  --timeout=100 [email protected]$host01::$dst --password-file=${rsync_passfile} >/dev/null 2>&1

done

exit 0

[[email protected] scripts]# sh inotify.sh &  #将脚本加入后台执行

[1] 13438

[[email protected] scripts]#

5.1.5 Real-time synchronization test

Inotify-master Operation:

[[email protected] scripts]# cd /backup/

[[email protected] backup]# ll

总用量 0

[[email protected] backup]# for a in `seq 200`;do touch $a;done  #创建200个文件

[[email protected] backup]# ll  --time-style=full-iso

总用量 0

-rw-r--r-- 1 root root 0 2014-04-22 15:34:08.141497569 +0800 1

-rw-r--r-- 1 root root 0 2014-04-22 15:34:08.172497529 +0800 10

-rw-r--r-- 1 root root 0 2014-04-22 15:34:08.235497529 +0800 100

-rw-r--r-- 1 root root 0 2014-04-22 15:34:08.236497529 +0800 101

-rw-r--r-- 1 root root 0 2014-04-22 15:34:08.237497529 +0800 102

...................................

Inotify-slave Check

[[email protected] backup]# ll  --time-style=full-iso

总用量 0

-rw-r--r--. 1 rsync rsync 0 2014-04-22 15:34:08.393823754 +0800 1

-rw-r--r--. 1 rsync rsync 0 2014-04-22 15:34:08.393823754 +0800 10

-rw-r--r--. 1 rsync rsync 0 2014-04-22 15:34:08.393823754 +0800 100

-rw-r--r--. 1 rsync rsync 0 2014-04-22 15:34:08.393823754 +0800 101

-rw-r--r--. 1 rsync rsync 0 2014-04-22 15:34:08.393823754 +0800 102

..........................

This article is from the "Chocolee" blog, make sure to keep this source http://chocolee.blog.51cto.com/8158455/1400596

2015-07-07 rsync+inotify for real-time synchronization