Rsync + inotify Real-Time Synchronization

Rsync Introduction

Rsync is a file synchronization and data transmission tool in Linux and UNIX systems. It uses the "rsync" algorithm to synchronize files between a client and a remote file server, unlike the limitations of cp and wget commands, it supports Incremental backup, so the file transmission efficiency is high, so the synchronization time is short. The specific features are as follows:

1. images can be used to save the entire directory tree and File System

2. Incremental data synchronization is supported. High file transmission efficiency.

3. You can maintain the permissions, time, and other attributes of the original file.

4. Encrypted data transmission to ensure data security.

5. You can use rcp, ssh, and other methods to transmit files.

Build a Remote Disaster Recovery and backup system

To ensure data security, the Web server backs up data to the remote disaster recovery server at every day. Due to the large data volume, only Incremental Backup can be performed each year. Only new data on the current day is backed up, the system environment is as follows:

Operating System RHEL5.8
Kernel version 2.6.18-308. el5web server address 192.168.1.104 Remote Disaster Recovery server address 192.168.1.110

Web configuration

Main configuration file/etc/rsyncd. conf

123456789101112131415161718192021 uid=nobody gid=nobody usechroot=no maxconnections=10 strictmodes= yes pid file = /var/run/rsyncd .pid lock file = /var/run/rsync .lock log file = /var/log/rsyncd .log [ixdba] path= /webdata commnet=ixdba file ignoreerrors read only=no writeonly=no hostsallow=* hostsdeny=192.168.1.200 list= false uid=root gid=root auth users =backup secrets file = /etc/server .pass

Password File chmod600/etc/server. pass

1 backup:ixdba123

Start the rsync daemon

1 /usr/local/bin/rsync --daemon

Remote Disaster Recovery Configuration

Password File chmod600/etc/server. pass

1 ixdba123

Add scheduled task crontab-e

1 13*** /usr/local/bin/rsync -vzrtopg--delete--progress--exclude "*access*" --exclude "debug*" backup@192.168.1.104::ixdba /ixdba . dir --password- file = /etc/server .pass

Insufficient Analysis

At this time, a remote disaster recovery system has been built, but this is not a complete disaster recovery solution, because rsync needs to be triggered to synchronize server data, therefore, the data on the server and client may be inconsistent during the two synchronization triggering intervals. If the website system encounters problems during this interval, the data will inevitably be lost, linux Kernel later than 2.6.13 provides the inotify File System Monitoring Mechanism, and uses the combination of rsync and inotify to achieve real-time data synchronization between the rsync server and the client.

Rsync + inotify Real-time Data Backup

Inotify is a powerful, fine-grained, asynchronous file system time monitoring mechanism. Since version 2.6.13, the Linux kernel has added support for inotify. Inotify can be used to monitor the addition, deletion, modification, movement, and other minor events in the file system. Using this kernel interface, third-party software can monitor various file changes in the file system, inotify-tools is a third-party software based on this requirement.

The Content Delivery Server synchronizes data to two Web servers in real time. inotify is a tool used to monitor file system changes. Therefore, it must be installed on the content publishing node, the content publishing node (Server) as the rsync client, two Web nodes act as the rsync server. The entire data synchronization process is a process of sending data from the client to the server, in contrast to the logic structure in the previous case, the system environment is as follows:

Node name kernel version IP address webpage data storage path Web12.6.18-308. el5192.168.1.110/web1/wwwrootWeb22.6.18-308. el5192.168.1.160/web2/wwwrootServer2.6.18-308. el5192.168.1.104/web/wwwroot

Web1 Configuration

Main configuration file/etc/rsyncd. conf

123456789101112131415161718192021 uid=nobody gid=nobody usechroot=no maxconnections=10 strictmodes= yes pid file = /var/run/rsyncd .pid lock file = /var/run/rsync .lock log file = /var/log/rsyncd .log [web1] path= /web1/wwwroot commnet=web1 file ignoreerrors read only=no writeonly=no hostsallow=192.168.1.104 hostsdeny=* list= false uid=root gid=root auth users =web1user secrets file = /etc/server .pass

Password File chmod600/etc/server. pass

1 web1user:ixdba123

Start the rsync daemon and add automatic start upon startup

12 echo "/usr/local/bin/rsync" >> /etc/rc . local /usr/local/bin/rsync--daemon

Web2 Configuration

Main configuration file/etc/rsyncd. conf

123456789101112131415161718192021 uid=nobody gid=nobody usechroot=no maxconnections=10 strictmodes= yes pid file = /var/run/rsyncd .pid lock file = /var/run/rsync .lock log file = /var/log/rsyncd .log [web2] path= /web2/wwwroot commnet=web2 file ignoreerrors read only=no writeonly=no hostsallow=192.168.1.104 hostsdeny=* list= false uid=root gid=root auth users =web2user secrets file = /etc/server .pass

Password File chmod600/etc/server. pass

1 web2user:ixdba123

Start the rsync daemon and add automatic start upon startup

12 echo "/usr/local/bin/rsync" >> /etc/rc . local /usr/local/bin/rsync--daemon

Server Configuration

Install inotify-tools

1234 tar xf rsync -3.0.7. tar .gz cd rsync -3.0.7 . /configure make && make install

Password File chmod600/etc/server. pass

1 ixdba123

Monitor directory changes and synchronize Web node scripts

123456789101112131415 #!/bin/sh host1=192.168.1.110 host2=192.168.1.160 dir = /web/wwwroot/ dst1=web1 dst2=web2 usr1=web1user usr2=web2user /usr/local/bin/inotifywait -mrq--timefmt '%d/%m/%y%H:%M' -- format '%T%w%f%e' -eclose_write,delete,create,attrib$ dir \ | while read files do /usr/bin/rsync -vzrtopg--delete--progress--password- file = /etc/server .pass$ dir $usr1@$host1::$dst1 /usr/bin/rsync -vzrtopg--delete--progress--password- file = /etc/server .pass$ dir $usr2@$host2::$dst2 echo "${files}wasrsynced" >> /tmp/rsync .log2>&1 done

Specify permissions and run them in the background

12 chmod 755 /web/wwwroot/inotifyrsync .sh /web/wwwroot/inotifyrsync .sh&

Add boot auto-start for this script

1 echo "/web/wwwroot/inotifyrsync.sh&" >> /etc/rc . local

Test Results

Server (rsync client) to create a test file,

Web1 view,

View on Web2,