Real-time synchronization of data in Linux under Rsync+inotify-tools

Address: http://www.osyunwei.com/archives/7435.html

Description

Operating system: CentOS 5.X

Source server: 192.168.21.129

Target server: 192.168.21.127,192.168.21.128

Objective: To synchronize the/home/www.osyunwei.com directory on the source server to the/home/www.osyunwei.com of the target server in real time

Specific operation:

The first part: operate on two target servers 192.168.21.127,192.168.21.128

One, respectively, on the target server to install the Rsync service side

1. Turn off SELinux

Vi/etc/selinux/config #编辑防火墙配置文件

#SELINUX =enforcing #注释掉

#SELINUXTYPE =targeted #注释掉

selinux=disabled #增加

: wq! #保存, Exit

Setenforce 0 #立即生效

2. Open Firewall TCP 873 port (rsync default port)

Vi/etc/sysconfig/iptables #编辑防火墙配置文件

-A rh-firewall-1-input-m state--state new-m tcp-p TCP--dport 873-j ACCEPT

: wq! #保存, Exit

/etc/init.d/iptables Restart #最后重启防火墙使配置生效

3. Install Rsync Server Software

Yum Install rsync xinetd #安装

Vi/etc/xinetd.d/rsync #编辑配置文件, set boot up rsync

Disable = no #修改为no

: wq! #保存退出

/etc/init.d/xinetd start #启动 (xinetd to manage rsync services in CentOS)

4. Create a rsyncd.conf configuration file

Vi/etc/rsyncd.conf #创建配置文件, add the following code

Log file =/var/log/rsyncd.log #日志文件位置, automatically generate this file when Rsync is started, without having to create it in advance

Pidfile =/var/run/rsyncd.pid #pid文件的存放位置

Lock file =/var/run/rsync.lock #支持max the connections parameter

Secrets file =/etc/rsync.pass #用户认证配置文件, which stores the user name and password, which is created later

MOTD file =/etc/rsyncd. MOTD #rsync启动时欢迎信息页面文件位置 (file content customization)

[Home_www.osyunwei.com] #自定义名称

Path =/home/www.osyunwei.com/ #rsync服务端数据目录路径

Comment = home_www.osyunwei.com #模块名称与 [home_www.osyunwei.com] Custom name is the same

UID = root #设置rsync运行权限为root

GID = root #设置rsync运行权限为root

port=873 #默认端口

Use chroot = no #默认为true, modify to No, increase backup of soft connection to directory file

Read Only = no #设置rsync服务端文件为读写权限

List = no #不显示rsync服务端资源列表

Max connections = #最大连接数

Timeout = #设置超时时间

Auth users = Home_www.osyunwei.com_user #执行数据同步的用户名, can be set multiple, separated by commas in the English state

Hosts allow = 192.168.21.129 #允许进行数据同步的客户端IP地址, can be set multiple, separated by commas in the English state

Hosts deny = 192.168.21.254 #禁止数据同步的客户端IP地址, can be set multiple, separated by commas in the English state

: wq! #保存, Exit

5. Create User authentication files

Vi/etc/rsync.pass #配置文件, add the following :

home_www.osyunwei.com_user:123456 #格式, user name: password, can be set multiple, one user name per line: password

: wq! #保存, Exit

6. Set file permissions

chmod 600/etc/rsyncd.conf #设置文件所有者读取, write permission

chmod 600/etc/rsync.pass #设置文件所有者读取, write permission

7. Start Rsync

/etc/init.d/xinetd Start #启动

Service xinetd Stop #停止

Service xinetd Restart #重新启动

Part Two: operating on the source server 192.168.21.129

First, install the rsync client

1. Turn off SELinux

Vi/etc/selinux/config #编辑防火墙配置文件

#SELINUX =enforcing #注释掉

#SELINUXTYPE =targeted #注释掉

selinux=disabled #增加

: wq! #保存, Exit

Setenforce 0 #立即生效

2, open the firewall TCP 873 Port (rsync default port, as the client rsync can not open 873 port)

Vi/etc/sysconfig/iptables #编辑防火墙配置文件

-A rh-firewall-1-input-m state--state new-m tcp-p TCP--dport 873-j ACCEPT

: wq! #保存, Exit

/etc/init.d/iptables Restart #最后重启防火墙使配置生效

System operation and maintenance www.osyunwei.com warm reminder: qihang01 original content copyright, reproduced please indicate the source and the original link

3. Installing the Rsync client software

Whereis rsync #查看系统是否已安装rsync, the following prompt appears stating that you have installed

Rsync:/usr/bin/rsync/usr/share/man/man1/rsync.1.gz

Yum Install xinetd #只安装xinetd即可, CentOS is a xinetd to manage rsync services

Yum Install rsync xinetd #如果默认没有rsync, run this command to install rsync and xinetd

Vi/etc/xinetd.d/rsync #编辑配置文件, set boot up rsync

Disable = no #修改为

/etc/init.d/xinetd start #启动(xinetd to manage rsync services in CentOS)

4. Create the authentication password file

Vi/etc/passwd.txt #编辑文件, add the following :

123456 #密码

: wq! #保存退出

chmod 600/etc/passwd.txt #设置文件权限, only set the file owner to have read, write access

5, the test source server 192.168.21.129 to two target server 192.168.21.127,192.168.21.128 data synchronization between

Mkdir/home/www.osyunwei.com/ceshi #在源服务器上创建测试文件夹 and then run the following 2-line command on the source server

RSYNC-AVH--port=873--progress--delete/home/www.osyunwei.com/[email protected]192.168.21.127::home_ Www.osyunwei.com--password-file=/etc/passwd.txt

RSYNC-AVH--port=873--progress--delete/home/www.osyunwei.com/[email protected]192.168.21.128::home_ Www.osyunwei.com--password-file=/etc/passwd.txt

After the completion of the operation, respectively on the two target server 192.168.21.127,192.168.21.128 view, in the /home/www.osyunwei.com directory has Ceshi folder, indicating that the data synchronization success.

Second, the installation of Inotify-tools tools, real-time trigger rsync synchronization

1. Check whether the server kernel supports inotify

Ll/proc/sys/fs/inotify #列出文件目录, the following content appears, stating that the server kernel supports inotify

-rw-r--r--1 root root 0 Mar 7 02:17 max_queued_events

-rw-r--r--1 root root 0 Mar 7 02:17 max_user_instances

-rw-r--r--1 root root 0 Mar 7 02:17 max_user_watches

Note: Linux supports inotify with a minimum kernel of 2.6.13, you can enter the command:uname-a view the kernel

The CentOS 5.X kernel is 2.6.18, INotify is already supported by default

2, Installation Inotify-tools

Yum install make gcc gcc-c++ #安装编译工具

Inotify-tools:http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz

Upload inotify-tools-3.14.tar.gz to the/USR/LOCAL/SRC directory

Cd/usr/local/src

Tar zxvf inotify-tools-3.14.tar.gz #解压

CD inotify-tools-3.14 #进入解压目录

./configure--prefix=/usr/local/inotify #配置

Make #编译

Make install #安装

3, set the system environment variable, add soft connection

echo "Path=/usr/local/inotify/bin: $PATH" >>/etc/profile.d/inotify.sh

source/etc/profile.d/inotify.sh #使设置立即生效

echo "/usr/local/inotify/lib" >/etc/ld.so.conf.d/inotify.conf

Ln-s/usr/local/inotify/include/usr/include/inotify

4. Modify inotify default parameter (inotify default kernel parameter value is too small)

View system default parameter values

sysctl-a | grep max_queued_events

The result is: Fs.inotify.max_queued_events = 16384

sysctl-a | grep max_user_watches

The result is: Fs.inotify.max_user_watches = 8192

sysctl-a | grep max_user_instances

The result is: fs.inotify.max_user_instances = 128

Modify Parameters:

Sysctl-w fs.inotify.max_queued_events= "99999999"

Sysctl-w fs.inotify.max_user_watches= "99999999"

Sysctl-w fs.inotify.max_user_instances= "65535"

vi/etc/sysctl.conf #添加以下代码

fs.inotify.max_queued_events=99999999

fs.inotify.max_user_watches=99999999

fs.inotify.max_user_instances=65535

: wq! #保存退出

Parameter description:

Max_queued_events:

INotify Queue Maximum Length, if the value is too small, a "* * Event Queue Overflow * * *" error occurs, resulting in inaccurate monitoring files

Max_user_watches:

How many directories are included in the file to be synchronized, by: Find/home/www.osyunwei.com-type D | Wc-l statistics, must ensure that the Max_user_watches value is greater than the statistical results (here/home/www.osyunwei.com to synchronize the file directory)

Max_user_instances:

Each user creates a INotify instance maximum value

System operation and maintenance www.osyunwei.com warm reminder: qihang01 original content copyright, reproduced please indicate the source and the original link

5, create a script, real-time trigger rsync synchronization

vi/usr/local/inotify/rsync.sh #编辑, add the following code

======================================

#!/bin/sh

srcdir=/home/www.osyunwei.com/

Dstdir=home_www.osyunwei.com

Rsyncuser=home_www.osyunwei.com_user

Rsyncpassdir=/etc/passwd.txt

dstip= "192.168.21.127 192.168.21.128"

For IP in $dstip

Do

RSYNC-AVH--port=873--progress--delete $srcdir [email protected] $ip:: $dstdir--password-file= $rsyncpassdir

Done

/usr/local/inotify/bin/inotifywait-mrq--timefmt '%d/%m/%y%h:%m '--format '%T%w%f%e '-e close_write,modify,delete,  Create,attrib,move $srcdir | While read file

Do

For IP in $dstip

Do

RSYNC-AVH--port=873--progress--delete $srcdir [email protected] $ip:: $dstdir--password-file= $rsyncpassdir

echo "${file} was rsynced" >>/tmp/rsync.log 2>&1

Done

Done

======================================

chmod +x/usr/local/inotify/rsync.sh #添加脚本执行权限

Script parameter Description:

srcdir=/home/www.osyunwei.com/ #源服务器同步目录

dstdir=home_www.osyunwei.com #目标服务器rsync同步目录模块名称

Rsyncuser=home_www.osyunwei.com_user #目标服务器rsync同步用户名

Rsyncpassdir=/etc/passwd.txt #目标服务器rsync同步用户的密码在源服务器的存放路径

dstip= "192.168.21.127 192.168.21.128" #目标服务器ip, multiple IPs separated by spaces

/tmp/rsync.log #脚本运行日志记录

6, set the script to start the automatic execution

Vi/etc/rc.d/rc.local #编辑, add a row at the end

Sh/usr/local/inotify/rsync.sh & # Set boot automatically run script in background

: wq! #保存退出

7, Test inotify real-time trigger rsync synchronization script is working

create a file on the source server 192.168.21.129 Inotify_rsync_ceshi

Mkdir/home/www.osyunwei.com/inotify_rsync_ceshi

Restart the source server:192.168.21.129

After the system starts, see If there are Inotify_rsync_ceshi folders under/home/www.osyunwei.com for both target servers 192.168.21.127,192.168.21.128

Then create the folder on the source server 192.168.21.129 inotify_rsync_ceshi_new

Mkdir/home/www.osyunwei.com/inotify_rsync_ceshi_new

Continue to see if there are inotify_rsync_ceshi_new folders under/home/www.osyunwei.com for both target servers 192.168.21.127,192.168.21.128

If all of the above tests pass, the inotify real-time trigger rsync synchronization script is running normally.

At this point, Linux rsync+inotify-tools implementation of real-time data synchronization completed.

Extended reading:

============================================

INotify parameters

-M is to keep listening

-R is a recursive view directory

-Q is the print out event

-e Create,move,delete,modify,attrib refers to the "Listen for create mobile Delete Write permission" event

rsync parameters

============================================

-V,--verbose verbose mode output

-Q,--quiet thin output mode

-C,--checksum turn on the check switch to force verification of file transfers

-A,--archive archive mode, which means to transfer files recursively and keep all file attributes equal to-rlptgod

-R,--recursive subdirectories in recursive mode

-R,--relative using relative path information

-B,--backup creates a backup, that is, the old file is renamed to ~filename when the same file name exists for the purpose. You can use the--suffix option to specify a different backup file prefix.

--backup-dir store backup files (such as ~filename) in the directory.

-suffix=suffix Defining backup File prefixes

-U,--update only updates, which is to skip all the files that already exist in DST, and the file time is later than the file to be backed up. (Does not overwrite the updated file)

-L,--links reserved Soft link

-L,--copy-links to handle soft links like regular files

--copy-unsafe-links only copies links to links outside the SRC Path directory tree

--safe-links ignoring links to the SRC Path directory tree

-H,--hard-links reserved Hard link

-P,--perms maintain file permissions

-O,--owner keep file owner information

-G,--group keep file group information

-D,--devices keep device file information

-T,--times keep file time information

-S,--sparse special processing of sparse files to save DST space

-N,--dry-run reality which files will be transmitted

-W,--whole-file copy files without incremental detection

-X,--one-file-system do not cross file system boundaries

-B, the block size used by the--block-size=size test algorithm, is 700 bytes by default

-E,--rsh=command specifies data synchronization using RSH and SSH

--rsync-path=path Specify the path information for the rsync command on the remote server

-C,--cvs-exclude uses the same method as CVs to automatically ignore files to exclude files that you do not want to transfer

--existing only updates those files that already exist in DST without backing up those newly created files

--delete Delete those files that are not in the DST SRC

--delete-excluded also deletes those files that are excluded by the option specified by the Receive side

--delete-after transfer ends after removal

--ignore-errors Timely IO errors are also deleted

--max-delete=num deleting NUM files up to a maximum

--partial retains files that are not fully transmitted for any reason, to expedite subsequent transmissions

--force forcibly delete a directory, even if it is not empty

--numeric-ids does not match the user and group ID of a number to a user name and group name

--timeout=time IP time-out, in seconds

-I,--ignore-times do not skip files that have the same time and length

--size-only when deciding whether to back up a file, just look at the file size regardless of file time

--modify-window=num determines whether the file is time-stamped with the time Stamp window, which defaults to 0

-t--temp-dir=dir creating temporary files in Dir

--compare-dest=dir also compares the files in DIR to determine if a backup is required

-p equivalent to--partial

--progress Show Backup process

-Z,--compress compress the backed-up files during transmission

--exclude=pattern specify to exclude file modes that do not need to be transferred

--include=pattern specifies file modes that need to be transferred without exclusion

--exclude-from=file exclude files in the specified schema in file

--include-from=file does not exclude files that specify pattern matching

--version Print version Information

--address binding to a specific address

--config=file specify a different configuration file, do not use the default rsyncd.conf file

--port=port Specify a different rsync service port

--blocking-io using blocking IO for remote shells

-stats gives the transfer status of some files

--progress in the transmission of the real-time transmission process

--log-format=format specifying the log file format

--password-file=file get the password from file

--bwlimit=kbps limit I/O bandwidth, Kbytes per second

-H,--help display Help information

Real-time synchronization of data in Linux under Rsync+inotify-tools