Rsync+inotify for real-time data synchronization

1. Introduction of environment and related software

Operating system: ubuntu12.04_x64

Source Server (push): 192.168.18.10

Target server (Backup): 192.168.18.20

Rsync:

Rsync is a data mirroring Backup tool under UNIX systems and is a fast incremental backup tool (remote sync) that supports local replication or synchronizes with other SSH (secure transfer) and rsync hosts.

Rsync has the following common parameters:

-v,--verbose Show details of the synchronization process

-a,--archive archive mode, which means that files are transferred recursively, and all file attributes are maintained, equivalent to-rlptgod

-r,--recursive Recursive mode for subdirectories

-z,--compress compressing the backed-up files while they are being transmitted

-l,--links Retaining Soft connections

-h,--hard-links Keep Hard Links

--delete Delete those files that are not in the DST src (source server Delete file, target server also deleted)

--progress shows the backup process, equivalent to-p

--port=port specifying the Rsync service port

--exclude=file exclude directories or files, multiple directories or files write multiple--exclude

INotify

INotify is a Linux feature that monitors file system operations such as read, write, and create, triggering inotify when a file system changes. INotify provides the Inotify-tools tool, which contains two functions, one is inotifywait to monitor file system changes, and the other is Inotifywatch, which is used to count the number of file system accesses. This time we use inotifywait with rsync to achieve real-time synchronization capabilities.

Inotifywait has the following common parameters:

-m,--monitor Keep Monitoring Events

-r,--recursive Recursive monitoring Directory

-q,--quiet Print-Triggered events

--timefmt specifying the time format for the%t format in-format selection

--FORMAT specifies the output format.

%w indicates the directory where the event occurred

%f indicates the file in which the event occurred

%e indicates that an event occurred

%T using the time format defined by-TIMEFMT

Inotifywait Common Listening Events:

Access file or directory read

Modify file or directory changes

attrib file or directory property changes

Move file or directory movement

Create file or directory creation

Delete file or directory deletion

2. Target server Installation and configuration

# sudo apt-get install rsync     #ubuntu系统默认已安装 # sudo cp/usr/ share/doc/rsync/examples/rsyncd.conf /etc/# sudo vi /etc/rsyncd.conf#  The following is the global configuration log  file = /var/log/rsyncd.logpid file = /var/run/rsyncd.pidlock file =  /var/lock/rsyncd# The following is the module option configuration [home]      #模块名, specify this name on the source server    comment  = sync rsync/home       #描述信息    path = / home/rsync       #备份目录    use chroot=no             #不使用chroot without root privileges    read only =  no           #设置本地备份目录为读写权限    uid=root                 gid=root    max connections=10        #客户端最大连接数    auth users = rsync        #指定数据同步用户    secrets file = /etc/rsyncd.pass            #指定数据同步用户信息文件    hosts allow= 192.168.18.0/24      #允许连接的客户端    ignore errors = yes       #忽略出现I/O error    timeout = 600

  #创建认证文件

# sudo vi /etc/rsyncd.passrsync:123456        #格式是用户名: Password # sudo chmod 600 /etc/rsyncd.pass        #属主要有权限读这个文件, otherwise will be reported not authorized # sudo /etc/init.d/rsync start     # If the startup reported the following error, then open the/etc/default/rsync file as prompted, will rsync_enable=false the rsync_enable=true, and then restart. * rsync daemon not enabled in/etc/default/rsync, not starting...# to see if it starts, There is an rsync listening port to indicate normal:# sudo netstat -antp |grep rsynctcp         0     0 0.0.0.0:873             0.0.0.0:*                listen      29605/rsync 

3. Source Server Installation and configuration

# sudo apt-get install rsync Inodify-tools#rsync is not configured because we only use the rsync command immediately after creating the authentication file # sudo vi/etc/rsyncd.pass123456 #只写密码 # s Udo Chmod 600/etc/rsyncd.pass

#此时基本配置完成, can you push it under test?

# RSYNC-AVZP--password-file=/etc/rsyncd.pass--delete/home/rsync [email protected]::home # Home is the module name of rsync on the target server sending incremental file listrsync/a 0 100% 0.00kb/s 0:00:00 (xfer#1, TO-CHECK=5/7) s ENT 354 Bytes received 126 bytes 960.00 bytes/sectotal size is 0 speedup is 0.00

#出现以上信息说明同步正常, Next, you should write the shell script, use inotifywait to do real-time monitoring source directory, add a while loop to determine if the source directory has a trigger, and if there is a change, perform rsync synchronization and log:

# vi inotify_rsync.sh#!/bin/bashsrc= '/home/rsync ' dst= ' [email protected]::home '/usr/bin/inotifywait-mrq--timefmt '% y-%m-%d%h:%m '--format '%T%w%f%e '-e create,delete,move,modify $SRC |while read FILESDO rsync-avzp--password-file= /etc/rsyncd.pass--delete $SRC $DST echo "$files was rsynced." >>/tmp/rsync.logdone

4. Test real-time synchronization

#先打印形式查看脚本执行情况

# bash -x inotify_rsync.sh + src=/home/rsync/          #目录结尾一定要加个/, otherwise the target server will be created again [email protected]::home+ read files+ /usr/bin/ inotifywait -mrq--timefmt  '%y-%m-%d %h:%m '  --format  '%t %w%f %e '  - ecreate,delete,move,modify,attrib /home/loongtao+ rsync -avzp --password-file=/etc/ rsyncd.pass --delete /home/rsync [email protected]@192.168.18.213::homesending  incremental file listrsync/rsync/test.txt            0 100%    0.00kB/s   0:00:00  (xfer#1, to-check=2 /11)  sent 349 bytes  received 32 bytes  762.00 bytes/ sectotal size is 9380  speedup is 24.62+ echo  ' 15-04-24  13:33/home/rsync/test.txt create was rsynced. ' + read files+ rsync -avzp--password-file=/etc/rsyncd.pass --delete /home/rsync  [email protected]::homesending incremental file list sent 310 bytes   received 10 bytes  640.00 bytes/sectotal size is 9380   speedup is 29.31+ echo  ' 15-04-24 13:33/home/rsync/test.txt attrib  was rsynced. ' + read files

#可以看到以上信息, without any error, indicates that it has been successfully pushed.

Then, put the script in the background to run:

# chmod +x inotify_rsync.sh

#./inotify_rsync.sh &

#此时当源服务器/home/rsync directory files are updated, they are synchronized to the target server/home/rsync directory

This article is from the "Penguin" blog, please be sure to keep this source http://lizhenliang.blog.51cto.com/7876557/1637926

Rsync+inotify for real-time data synchronization