Data synchronization with Rsync+inotify
I. Rsync concept:
Rsync is a tool for file synchronization and data transmission under Linux system, using the rsync algorithm to make the client and server, the master server and
Backup server data Synchronization. Rsync can also resume transmission after an outage. Rsync supports incremental backups.
Two. Rsync mode:
1. Local mode.
2. Remote Shell mode
3. Query mode
4.c/s mode
To install Rsync:
Tar xvf rsync-3.0.9.tar.gz
CD rsync-3.0.9./configure
Make && make install
rsync--help Viewing options
-V,--verbose//verbose mode
-R,--recursive//recursion
-U,--update//update
-T,--times//Hold Time
-Z,--compress//enable compression
-O,--owner//maintain owner
-G,--group//retention Group
-P,--perms//Maintain permissions
--delete//previous Edge directory, synchronization
--progress//Display transfer process
--exclude=pattern//file with different step pattern matching
--password-file=file//Password file location
1. Local mode
RSYNC-RV/ETC/PASSWD. Copy passwd to current directory
2. Remote Shell mode
RSYNC-RV/ETC/PASSWD 172.18.0.1:/tmp//Copy the passwd to the remote host
3. List mode
Rsync-r 172.18.0.6:/tmp # #查看远程目录下的文件
All of the above three modes can be implemented using SSH.
Planning:
master:172.18.1.1
slave:172.16.1.1
Install the configuration rsync on master
1.1 Installation
Tar xvf rsync-3.0.9.tar.gz
CD rsync-3.0.9./configure
Make && make install
1.2 Configuring Rsync
Installation-generated configuration file sample under/usr/local/share/man/man5, is rsyncd.conf.5;
Copy to/etc/the rsyncd.conf;rsyncd.conf file consists of a number of modules, including global parameters and module parameters.
The main contents are:
UID = nobody//global configuration starts, refers to the UID of the module process when the file is transferred
GID = nobody//Ibid. gid
Use chroot = no//whether to let the process leave the working directory
Max connections = 4//MAX concurrent number
Syslog facility = LOCAL5//logging facility
PID file =/var/run/rsyncd.pid//pid position
[FTP]//module configuration start
Path =/var/ftp/pub//directory to be backed up, must be specified,
Comment = Whole ftp area//Comment
Read Only = no//client is read-only
Write only = no//write only
Hosts allow = *//allows synchronization of hosts
Hosts deny = 192.168.0.0/24//Forbidden Host
List = yes/allow all modules to be listed
UID = root
GID = root
Auth users = slave//user who can connect to the module
Secrets file =/etc/rsync.pass//password files where you need to build it yourself
1.3 Create a password file/etc/rsync.pass the following format, and ensure that the permissions are (rw-------) or (R--------)
Slave:helloworld
1.4 Starting the Daemon
rsync--daemon # #启动守护进程
NETSTAT-TLNP |grep rsync to see if it starts, view the listening port
Third, slave end rsync settings
3.1 Installing Rsync
Tar xvf rsync-3.0.9.tar.gz
CD rsync-3.0.9./configure
Make && make install
3.2 Do not need the configuration file, directly run the command, in order to not enter the password, create a password file, and ensure that the permissions of 600
echo "HelloWorld" >/root/rsync.pass.slave
3.3 rsync has too many options, we write it in a script and run it when needed
vi/root/rsync.sh
#!/bin/bash
/USR/LOCAL/BIN/RSYNC-VZRTOGPG--delete--progress \
[Email protected]::ftp/var/ftp/pub--password-file=/root/rsync.pass.slave
chmod +x/root/rsync.sh//Modify Permissions
Iv. Testing
1. Add files to the master directory to see if you can sync
Cp/etc/passwd/var/ftp/pub
Slave run a script test
sh/root/rsync.sh
See if there are a few more files in the/var/ftp/pub
2. The client creates a backup policy that synchronizes every 5 seconds
Cron can only be accurate to the point, with the script to achieve every 5 seconds synchronization
VI cron.sh
#!/bin/bash
For ((i=1;i<=12;i++));d o
echo "*/1 * * * */root/rsync.sh" >>/var/spool/cron/root
Sleep 5
Done
Add the files again in the/var/ftp/pub to see if you can sync properly.
This way the disadvantage is that the client must synchronize every 5 seconds, so that the frequency is a waste of resources, the interval is long, the data can not be consistent, rsync each synchronization needs than the file changes, if more files, this is very inefficient, so we have to use a more efficient method, that is inotify.
3, INotify concept?
INotify is a powerful, fine-grained asynchronous file system event monitoring mechanism. Through the inotify can monitor the file system additions, deletions, modifications and so on, using this kernel interface, third-party software can monitor changes in the file system, thus triggering the synchronization of rsync, we use inotify-tools to achieve this function.
Idea: Inotify-tool should be installed on the host that needs to be monitored, if we install it on Master, monitor its own file system, when the file system changes, it should notify the client to synchronize, such as:
SSH 172.16.1.3 '/root/rsync.sh '
The premise of dual-machine SSH trust.
There is another way may be better than this, the inotify-tool installed on the slave, to monitor the slave file system, when the file system changes in the slave, the data sync to master up, but I think the role of the two reversed.
Five, method one:
Install Inotify-tools on 5.1 master
Download: http://cloud.github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz
Tar xvf inotify-tools-3.14.tar.gz
CD inotify-tools-3.14./configure
Make && make install
5.2 Generated two execution program Usr/local/bin/inotifywait/usr/local/bin/inotifywatch, inotifywait used to monitor file system changes, Inotifywatch used to statistical changes The system event.
Some parameters of 5.3 inotifywait
-M--monitor//Always Monitor
-R--recursive//recursive
-Q--quiet//Print monitoring Events
-e--event//Indicates the events to be monitored, including: modify,delete,create,attrib, etc.
5.4 Run Inotifywait-mrq--timefmt '%d/%m/%y%h:%m '--format '%T%w%e '-e modify,delete,create,attrib/var/ftp/pub toward/var/ft Add a file to the p/pub to see if there is any output, and if so, it means everything is OK.
--TIMEFMT Time Format
--format details of the change file
5.5 Write a script to achieve, when the/var/ftp/pub/Chinese pieces have changed, let slave sync
VI inotify_slave.sh
#!/bin/bash
Inotifywait-mrq--timefmt '%d/%m%y%h%m '--format '%T%w%f%e ' \
-E modify,delete,create,attrib/var/ftp/pub | While read files
Do
SSH 172.16.1.3 '/root/rsync.sh '//dual-machine mutual trust has been done
Done
5.6 Testing
Run the script to add a file test in/var/ftp/pub
SH inotify_slave.sh &
Cp-r/etc/rc.d/init.d/var/ftp/pub
See if the files in the slave are synchronized
Vi. method Two, install the inotify_tools on the slave
6.1 Installation
Download: http://cloud.github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz
Tar xvf inotify-tools-3.14.tar.gz
CD inotify-tools-3.14./configure
Make && make install
6.2 End script running on master
6.3 Write a script to monitor slave's/var/ftp/pub, when the/var/ftp/pub file system changes, immediately sync to master, so slave became the real master,master into slave
VI inotify_to_master.sh
#!/bin/bash
Inotifywait-mrq--timefmt '%d/%m%y%h%m '--format '%T%w%f%e ' \
-E modify,delete,create,attrib/var/ftp/pub/| While read files
Do
RSYNC-VZRTOGPG--delete--progress/var/ftp/pub [email protected]::ftp \
--password-file=/root/pass.rsync
Done
6.4 Add files to Slave's/var/ftp/pub, run the test
SH inotify_to_master.sh &
Cp-r/etc/yum/var/ftp/pub
See if sync is in master.
Data synchronization with Rsync+inotify