Rsync+inotify-tools implementing real-time synchronization of files

About Rsync:

Rsync is a remote data synchronization tool that uses the so-called "Rsync algorithm" to synchronize files between local and remote two hosts, which transmits only the different portions of two files, rather than the entire transfer every time, so it is quite fast. The machine running Rsync server is also called Backup server, a rsync server can back up data from multiple clients at the same time, or multiple rsync servers can back up one client's data. Rsync can be paired with rsh or SSH or even use daemon mode. Rsync server will open a 873 service channel (port), waiting for the other rsync connection. When connected, Rsync server checks that the password matches and, if checked with a password, can start the file transfer. When the first connection is complete, the entire file is transmitted once, and the next time only two files are transferred in different parts.

Inotify-tools Introduction:

Inotify-tools is a set of C's Development interface library functions for Linux under the INotify file Monitoring tool, as well as a series of command-line tools that can be used to monitor file system events. Inotify-tools is written in C, except that it requires the kernel to support inotify, and does not depend on others. Inotify-tools provides two tools, one is inotifywait, it is used to monitor file or directory changes, and the second is Inotifywatch, it is used to count the number of file system access. Now, let's introduce how it's used.

Environment deployment:

Inotify_slave 42.159.246.48:57893 data back-up

Inotify_master 42.159.246.48:57892 Source Data port

First, deploy the data backup side:

1. The backup side deploys the Rsync service:

[[Email protected] ~] #yum-y install rsync

[[Email protected] ~] #useradd rsync-s/sbin/nologin-m

[[Email protected] ~] #mkdir-P/cmsresoure/image/backup

[[Email protected] ~] #chown rsync:rsync/cmsresoure/image/backup

2. Add a configuration file/etc/rsyncd.conf

[[Email protected] ~] #vim/etc/rsyncd.conf

##################################################################################

# #rsyncd. conf start##

UID = rsync

GID = rsync

Use chroot = no

Max connections = 200

Timeout = 300

PID file =/var/run/rsyncd.pid

Lock file =/var/run/rsync.lock

Log file =/var/log/rsyncd.log

[Backup]

Path =/cmsresoure/image/backup/

Ignore errors

Read Only = False

List = False

Hosts allow = *

Hosts Deny = 0.0.0.0/32

Auth users = Rsync_backup

Secrets file =/etc/rsync.password

#rsync_config_______________end

##################################################################################

3. Configure the virtual user's password file and start the service

[Email protected] ~]# echo "Rsync_backup:leesir" >/etc/rsync.password

[Email protected] ~]# chmod 600/etc/rsync.password

[Email protected] ~]# rsync--daemon

[Email protected] ~]# Ps-ef | grep rsync

4. Test push via source data end

[[email protected] ~]# echo "Leesir" >/etc/rsync.password #注意: Here just write the password, remember.

[Email protected] ~]# chmod 600/etc/rsync.password

[Email protected] ~]# RSYNC-AVZLH inotify.sh [email protected]::backup--password-file=/etc/rsync.password

After the source data is pushed, go to the data backup side to check whether the push is successful.

Second, the source data-side deployment

1. Check whether the current system supports inotify

[Email protected] ~]# ll/proc/sys/fs/inotify/

Total 0

-rw-r--r--1 root root 0 Oct 8 18:54 max_queued_events

-rw-r--r--1 root root 0 Oct 8 18:54 max_user_instances

-rw-r--r--1 root root 0 Oct 8 18:54 max_user_watches

Remarks:

#显示这三个文件则证明支持.

/proc/sys/fs/inotify/max_queued_evnets

The maximum value of the number of events that can be queued #表示调用inotify_init时分配给inotify The instance, an event that exceeds this value is discarded, but the In_q_overflow event is triggered.

/proc/sys/fs/inotify/max_user_instances

#表示每一个real the maximum number of inotify instatnces that the user ID can create.

/proc/sys/fs/inotify/max_user_watches

#表示每个inotify instatnces The maximum number of directories that can be monitored. If you monitor a large number of files, you need to increase the size of this value appropriately, depending on the situation.

Example: Echo 30000000 >/proc/sys/fs/inotify/max_user_watches

2. Deploy the INotify service:

[Email protected] ~]# yum-y install Inotify-tools

[Email protected] ~]# Rpm-qa | grep inotify-tools

Inotify-tools-3.14-1.el6.x86_64

[Email protected] ~]# vim inotify.sh

##################################################################################

#!/bin/bash

#

#2015.10.8

#

desthost=42.159.246.48

srcdir=/cmsresoure/image/backup/

Passwd=/etc/rsync.password

User=rsync_backup

INOTIFYWAIT-MR--timefmt '%d/%m/%y%h:%m '--format '%T%w%f '-E close_write,modify,create,attrib $SRCDIR | While read DATE time DIR File;do

Filechange=${dir}${file}

RSYNC-AVZTOPGLH $SRCDIR [email protected]${desthost}::backup--password-file= $PASSWD &>/dev/null && \

echo "at ${time} on ${date}, file $FILECHANGE is backed up via rsync" >>/var/log/image.log

Done

##################################################################################

3. Test script to create a file test:

[Email protected] ~]# more/home/user/touch_file.sh

##################################################################################

#!/bin/sh

#

#2015.10.8

#

#Create 5 100K files, file name is a random number

#

#

For ((i=1;i<6;i++))

Do

DD If=/dev/zero of=/cmsresoure/image/backup/$ (sha1sum<<< $RANDOM |cut-b 5-10) bs=100k count=1

echo "Touch $i Done"

Sleep 3

Done

echo "The script is done"

##################################################################################

[[email protected] ~]# echo "*/30 * * * */bin/bash/home/user/touch_file.sh >/dev/null 2>&1" >>/etc/cro Nteb

[Email protected] ~]# sh inotify.sh & #将脚本加入后台执行

Setting up watches. Beware:since-r is given, this could take a while!

Watches established.

Three. Real-time Sync test

1. Source Data side operation:

[Email protected] ~]# sh/home/user/touch_file.sh

##################################################################################

1+0 Records in

1+0 Records out

102400 bytes (102 kB) copied, 0.000441594 s, 232 MB/s

Touch 1 Done

1+0 Records in

1+0 Records out

102400 bytes (102 kB) copied, 0.000603492 s, + MB/s

Touch 2 Done

1+0 Records in

1+0 Records out

102400 bytes (102 kB) copied, 0.000548994 s, 187 MB/s

Touch 3 Done

1+0 Records in

1+0 Records out

102400 bytes (102 kB) copied, 0.000423695 s, 242 MB/s

Touch 4 Done

The script is done

##################################################################################

[Email protected] ~]# ls/cmsresoure/image/backup/

039f90 15aaf2 1c2d4b 3b079d

2. Data Backup side check:

[Email protected] ~]# cd/cmsresoure/image/backup/

[[email protected] backup]# ls

039f90 15aaf2 1c2d4b 3b079d

Four. Maintenance:

1. Set up a script to monitor the Rsync service on the backup side:

[Email protected] shell]# vim rsync_check.sh

##################################################################################

#!/bin/bash

#

#2015.10.8

#

#The script would check the Rsync service, if you find that rsync would restart the service.

rsync_check= ' NETSTAT-NPL | Grep-c rsync '

If ["$RSYNC _check"-eq "0"];

Then

/usr/bin/rsync--daemon

Fi

##################################################################################

[[email protected] shell]# echo "*/2 * * * */bin/bash/shell/rsync_check.sh >/dev/null 2>&1" >>/etc/cron tab

Remarks:

Add the script to the scheduled task, not two minutes to check if Rsync is working

2. Set the script to check the INotify service on the source data side:

[Email protected] shell]# vim check_inotify.sh

##################################################################################

#!/bin/bash

#

#2015.10.9

#

#the script would to check the inotify server

#

#

inotify=/shell/inotify.sh

Check_inotify= ' Ps-ef | Grep-c inotify.sh '

If ["$CHECK _inotify"-eq "1"];

Then

/bin/bash "$INOTIFY" >/dev/null 2>&1

Fi

##################################################################################

[[email protected] shell]# echo "*/2 * * * */bin/bash/shell/check_inotify.sh >/dev/null 2>&1" >>/etc/ Crontab

Five. Test results:

Rsync+inotify-tools Sync Effect: rsync synchronizes the monitored folders in full real-time and outputs synchronization logs.

This article from "aestheticism" blog, declined reprint!

Rsync+inotify-tools implementing real-time synchronization of files