Use rsync + inotify for Data Synchronization

Using rsync + inotify for Data Synchronization I. What is rsync? It is a tool for file synchronization and data transmission in Linux. It uses the rsync algorithm to enable client and server, master server and
Backup server data synchronization. Rsync can also resume transmission after interruption. Rsync supports Incremental backup. Www.2cto.com 2. rsync has four modes: 1. Local Mode. 2. Remote shell Mode 3. query mode 4.C/ S. Let's install rsync and test it. Install the very simple tar xvf rsync-3.0.9.tar.gz cd rsync-3.0.9. /configure make & make install rsync -- help view option-v, -- verbose # detailed mode-r, -- recursive # recursive-u, -- update # update-t, -- times ## retention time-z, -- compress # enable compression-o, -- owner # Keep owner-g, -- group # Keep group-p, -- perms # retain permissions -- delete # The former edge directory prevails, synchronization -- progress # display the transmission process -- exclude = PATTERN # files in non-synchronous mode -- password-file = FILE # password file Location 1. local Mode rsync-rv/etc/passwd. # copy passwd to the current directory 2. remote sh Ell mode rsync-rv/etc/passwd 172.16.1.3:/tmp # copy passwd to remote host 3. list mode rsync-r 172.16.1.3:/tmp # The first three modes of viewing files in the remote directory can be implemented using ssh. We will study the fourth www.2cto.com plan: Master: 172.16.1.2Slave: 172.16.1.2 1. install and configure rsync1.1 on the Master node. Just now, you can configure rsync to generate a configuration file sample in/usr/local/share/man/man5, rsyncd. copy conf.5 to/etc/and call it rsyncd. conf, rsyncd. the conf file consists of multiple modules, including global parameters and module parameters. The previous part is a comment. The main content is: uid = nobody # Start with global configuration, indicates the uid gid = nobody of the module process during file transfer. # Same as gid use chroot = no # whether to let the process out of the working directory max connections = 4 # maximum number of concurrent syslog facility = local5 # The facility pid file that records logs =/var /run/rsyncd. pid ## pid location [ftp] ## module configuration start path =/var/ftp/pub # directory to be backed up, which must be specified, comment = whole ftp area ## comment read only = no ## whether the client is read-only write only = no ## whether only hosts allow = * ## allow synchronous host hosts deny = 192.168.0.0/ 24 ## list of hosts with access prohibited = yes ## list all modules uid = root gid = root auth users = sl Ave ## user secrets file =/etc/rsync that can be connected to this module. pass ## where is the password file? Create your own 1.3 password file/etc/rsync. pass the following format and make sure the permission is 600 or 400 slave: helloworld 1.4 to start the rsync -- daemon # Start the daemon process netstat-tlnp | grep rsync to check whether the daemon process is started, check the listening port 2. Set rsync on the slave to 2.1 to install rsync. Check that the above 2.2 does not require a configuration file. Run the command directly. In order not to enter a password, create a password file, and make sure the permission is 600 echo "helloworld">/root/rsync. pass. slave 2.3 rsync has too many options. We can write it into a script and run vi/root/rsync as needed. sh #! /Bin/bash/usr/local/bin/rsync-vzrtogpg -- delete -- progress \ slave@172.16.1.2: ftp/var/ftp/pub -- password-file =/root/rsync. pass. slave chmod + x/root/rsync. sh ## Modify permissions 3. Test adding files to the master Directory, check whether cp/etc/passwd/var/ftp/pub slave can be synchronized and run the script to test sh/root/rsync. sh check whether there are multiple files in/var/ftp/pub. 4. Create a backup policy on the client and synchronize data every five seconds. We all know that cron can only be accurate to minutes, we will use a script to synchronize vi cron every five seconds. sh #! /Bin/bash for (I = 1; I <= 12; I ++); do echo "*/1 *****/root/rsync. sh ">/var/spool/cron/root sleep 5 done Add a file to/var/ftp/pub again to check whether the file can be synchronized normally. The disadvantage of this method is that the client must be synchronized every five seconds. This frequency is a waste of resources. If the interval is long, data cannot be consistent, rsync needs to compare changes to files during each synchronization. If there are more files, this is very inefficient. Therefore, we need to use a more efficient method, that is inotify. What is inotify? Inotify is a powerful fine-grained asynchronous file system event monitoring mechanism. Inotify can be used to monitor the addition, deletion, and modification of file systems. Using this kernel interface, third-party software can monitor the changes in the file system and trigger rsync synchronization operations, we use inotify-tools to implement this function. Idea: inotify-tool should be installed on the host to be monitored. If we install it on the master to monitor our own file system, when the file system changes, it should notify the client for synchronization, for example, ssh 172.16.1.3 '/root/rsync. sh' requires mutual trust over two-host ssh servers. Another method may be better. Install inotify-tool on slave to monitor the slave file system. When the file system in slave changes, I will synchronize my data to the master, but I really want to change the roles of the two. 5. Try the first idea 5.1 install unzip y-tools on the master download: http://cloud.github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz tar xvf inotify-tools-3.14.tar.gz cd inotify-tools-3.14. /configure make & make install 5.2 generates two execution programs: usr/local/bin/inotifywait/usr/local/bin/inotifywatch. inotifywait is used to monitor changes to the file system, inotifywatch is used to count events that change the file system. 5.3 inotifywait parameters-m -- monitor # Always monitoring-r -- recursive # recursive-q -- quiet # print monitoring event-e -- event # indicate event, include: modify, delete, create, attrib, etc. 5.4 run inotifywait-mrq -- timefmt '% d/% m/% y % H: % m' -- format' % T % w % e'-e modify, delete, create, attrib/var/ftp/pub Add a file to/var/ftp/pub, check whether output exists. If yes, everything is normal. -- Timefmt time format -- detailed information of the format change file 5.5 write a script implementation. When the file in/var/ftp/pub/changes, let slave synchronize vi inotify_slave.sh #! /Bin/bash inotifywait-mrq -- timefmt '% d/% m % y % H % m' -- format' % T % w % f % E' \-e modify, delete, create, attrib/var/ftp/pub | while read files do ssh 172.16.1.3 '/root/rsync. sh' # dual-host mutual trust has completed the done 5.6 test run script. Add a file in/var/ftp/pub to test sh inotify_slave.sh & cp-R/etc/rc. d/init. d/var/ftp/pub check whether the files in slave are synchronized, install inotify_tools6.1 on slave. The installation will not go into details. 6.2 end the script running on the master. 6.3 write a script to monitor slave/var/ftp/pub, when the/var/ftp/pub file system changes Immediately synchronize to the master, so that slave becomes the real master, and the master becomes slavevi inotify_to_master.sh #! /Bin/bash inotifywait-mrq -- timefmt '% d/% m % y % H % m' -- format' % T % w % f % E' \-e modify, delete, create, attrib/var/ftp/pub/| while read files do rsync-vzrtogpg -- delete -- progress/var/ftp/pub slave@172.16.1.2 :: ftp \ -- password-file =/root/pass. rsyncdone 6.4 adds a file to slave's/var/ftp/pub and runs the test sh inotify_to_master.sh & cp-R/etc/yum/var/ftp/pub to check whether the master is synchronized. conclusion: After the rsync experiment is completed, rsync is useful for data backup, especially for companies with insufficient funds, which can replace shared storage. So try it if you have time.