Data synchronization with Rsync+inotify

Data synchronization with Rsync+inotify

I. Rsync concept:

Rsync is a tool for file synchronization and data transmission under Linux system, using the rsync algorithm to make the client and server, the master server and

Backup server data Synchronization. Rsync can also resume transmission after an outage. Rsync supports incremental backups.

Two. Rsync mode:

1. Local mode.

2. Remote Shell mode

3. Query mode

4.c/s mode

To install Rsync:

Tar xvf rsync-3.0.9.tar.gz

CD rsync-3.0.9./configure

Make && make install

rsync--help Viewing options

-V,--verbose//verbose mode

-R,--recursive//recursion

-U,--update//update

-T,--times//Hold Time

-Z,--compress//enable compression

-O,--owner//maintain owner

-G,--group//retention Group

-P,--perms//Maintain permissions

--delete//previous Edge directory, synchronization

--progress//Display transfer process

--exclude=pattern//file with different step pattern matching

--password-file=file//Password file location

1. Local mode

RSYNC-RV/ETC/PASSWD. Copy passwd to current directory

2. Remote Shell mode

RSYNC-RV/ETC/PASSWD 172.18.0.1:/tmp//Copy the passwd to the remote host

3. List mode

Rsync-r 172.18.0.6:/tmp # #查看远程目录下的文件

All of the above three modes can be implemented using SSH.

Planning:

master:172.18.1.1

slave:172.16.1.1

Install the configuration rsync on master

1.1 Installation

Tar xvf rsync-3.0.9.tar.gz

CD rsync-3.0.9./configure

Make && make install

1.2 Configuring Rsync

Installation-generated configuration file sample under/usr/local/share/man/man5, is rsyncd.conf.5;

Copy to/etc/the rsyncd.conf;rsyncd.conf file consists of a number of modules, including global parameters and module parameters.

The main contents are:

UID = nobody//global configuration starts, refers to the UID of the module process when the file is transferred

GID = nobody//Ibid. gid

Use chroot = no//whether to let the process leave the working directory

Max connections = 4//MAX concurrent number

Syslog facility = LOCAL5//logging facility

PID file =/var/run/rsyncd.pid//pid position

[FTP]//module configuration start

Path =/var/ftp/pub//directory to be backed up, must be specified,

Comment = Whole ftp area//Comment

Read Only = no//client is read-only

Write only = no//write only

Hosts allow = *//allows synchronization of hosts

Hosts deny = 192.168.0.0/24//Forbidden Host

List = yes/allow all modules to be listed

UID = root

GID = root

Auth users = slave//user who can connect to the module

Secrets file =/etc/rsync.pass//password files where you need to build it yourself

1.3 Create a password file/etc/rsync.pass the following format, and ensure that the permissions are (rw-------) or (R--------)

Slave:helloworld

1.4 Starting the Daemon

rsync--daemon # #启动守护进程

NETSTAT-TLNP |grep rsync to see if it starts, view the listening port

Third, slave end rsync settings

3.1 Installing Rsync

Tar xvf rsync-3.0.9.tar.gz

CD rsync-3.0.9./configure

Make && make install

3.2 Do not need the configuration file, directly run the command, in order to not enter the password, create a password file, and ensure that the permissions of 600

echo "HelloWorld" >/root/rsync.pass.slave

3.3 rsync has too many options, we write it in a script and run it when needed

vi/root/rsync.sh

#!/bin/bash

/USR/LOCAL/BIN/RSYNC-VZRTOGPG--delete--progress \

[Email protected]::ftp/var/ftp/pub--password-file=/root/rsync.pass.slave

chmod +x/root/rsync.sh//Modify Permissions

Iv. Testing

1. Add files to the master directory to see if you can sync

Cp/etc/passwd/var/ftp/pub

Slave run a script test

sh/root/rsync.sh

See if there are a few more files in the/var/ftp/pub

2. The client creates a backup policy that synchronizes every 5 seconds

Cron can only be accurate to the point, with the script to achieve every 5 seconds synchronization

VI cron.sh

#!/bin/bash

For ((i=1;i<=12;i++));d o

echo "*/1 * * * */root/rsync.sh" >>/var/spool/cron/root

Sleep 5

Done

Add the files again in the/var/ftp/pub to see if you can sync properly.

This way the disadvantage is that the client must synchronize every 5 seconds, so that the frequency is a waste of resources, the interval is long, the data can not be consistent, rsync each synchronization needs than the file changes, if more files, this is very inefficient, so we have to use a more efficient method, that is inotify.

3, INotify concept?

INotify is a powerful, fine-grained asynchronous file system event monitoring mechanism. Through the inotify can monitor the file system additions, deletions, modifications and so on, using this kernel interface, third-party software can monitor changes in the file system, thus triggering the synchronization of rsync, we use inotify-tools to achieve this function.

Idea: Inotify-tool should be installed on the host that needs to be monitored, if we install it on Master, monitor its own file system, when the file system changes, it should notify the client to synchronize, such as:

SSH 172.16.1.3 '/root/rsync.sh '

The premise of dual-machine SSH trust.

There is another way may be better than this, the inotify-tool installed on the slave, to monitor the slave file system, when the file system changes in the slave, the data sync to master up, but I think the role of the two reversed.

Five, method one:

Install Inotify-tools on 5.1 master

Download: http://cloud.github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz

Tar xvf inotify-tools-3.14.tar.gz

CD inotify-tools-3.14./configure

Make && make install

5.2 Generated two execution program Usr/local/bin/inotifywait/usr/local/bin/inotifywatch, inotifywait used to monitor file system changes, Inotifywatch used to statistical changes The system event.

Some parameters of 5.3 inotifywait

-M--monitor//Always Monitor

-R--recursive//recursive

-Q--quiet//Print monitoring Events

-e--event//Indicates the events to be monitored, including: modify,delete,create,attrib, etc.

5.4 Run Inotifywait-mrq--timefmt '%d/%m/%y%h:%m '--format '%T%w%e '-e modify,delete,create,attrib/var/ftp/pub toward/var/ft Add a file to the p/pub to see if there is any output, and if so, it means everything is OK.

--TIMEFMT Time Format

--format details of the change file

5.5 Write a script to achieve, when the/var/ftp/pub/Chinese pieces have changed, let slave sync

VI inotify_slave.sh

#!/bin/bash

Inotifywait-mrq--timefmt '%d/%m%y%h%m '--format '%T%w%f%e ' \

-E modify,delete,create,attrib/var/ftp/pub | While read files

Do

SSH 172.16.1.3 '/root/rsync.sh '//dual-machine mutual trust has been done

Done

5.6 Testing

Run the script to add a file test in/var/ftp/pub

SH inotify_slave.sh &

Cp-r/etc/rc.d/init.d/var/ftp/pub

See if the files in the slave are synchronized

Vi. method Two, install the inotify_tools on the slave

6.1 Installation

Download: http://cloud.github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz

Tar xvf inotify-tools-3.14.tar.gz

CD inotify-tools-3.14./configure

Make && make install

6.2 End script running on master

6.3 Write a script to monitor slave's/var/ftp/pub, when the/var/ftp/pub file system changes, immediately sync to master, so slave became the real master,master into slave

VI inotify_to_master.sh

#!/bin/bash

Inotifywait-mrq--timefmt '%d/%m%y%h%m '--format '%T%w%f%e ' \

-E modify,delete,create,attrib/var/ftp/pub/| While read files

Do

RSYNC-VZRTOGPG--delete--progress/var/ftp/pub [email protected]::ftp \

--password-file=/root/pass.rsync

Done

6.4 Add files to Slave's/var/ftp/pub, run the test

SH inotify_to_master.sh &

Cp-r/etc/yum/var/ftp/pub

See if sync is in master.

Data synchronization with Rsync+inotify