Enterprise Case rsync+inotify Synchronizing Web data

Application Scenarios:

The company backend has two Nginx server, because still in the testing phase. So development often needs to update the Web program, and each update will need to upload two times. Now the two Web servers are OK. If in the future with the volume of business growth, 4, 5 sets of time also need to upload a bit bitter force. Therefore, I intend to use the rsync+inotify combination to synchronize the data of the server.

Rsync Introduction:

Rsync is a file synchronization and data Transfer tool under the Linux/unix system, which uses the rsync algorithm to synchronize files between a client and a remote file server. By using rsync, you can back up data from one partition to another on the same server, or you can back up your local system's data over a network to any remote host, and rsync can resume transmission after an outage, transferring only the inconsistent portions of the source and destination files. You can perform a full or incremental backup. Rsync has a shell (local mode), a remote shell, a query (a list, similar to what the LS command implements), and server mode.

Features of Rsync:

The entire directory tree and file system can be saved in a mirror.

Can incrementally synchronize data, file transfer efficiency is high, so synchronization time is very short

Can keep the original file permissions, time and other properties

Encrypt and transmit data to ensure the security of data

You can transfer files using RCP, SSH, and so on, or you can transfer files directly through the socket connection.

Support for anonymous transmissions

First, the installation of rsync:

I need to synchronize the data between the two Web servers, 10.171.22.124 to 10.172.186.241. Therefore, 124 is called the client, and 241 is called the server side.

1.1 Server-side installation:

Using Yum installation and source installation can be, I choose the source code.

Cd/usr/local/src

wget https://download.samba.org/pub/rsync/src/rsync-3.0.4.tar.gz

TAR-ZXVF rsync-3.0.4.tar.gz

CD rsync-3.0.4

./configure--prefix=/usr/local/rsync

Make && make install

Mkdir/etc/rsyncd

Touch/etc/rsyncd/rsyncd.conf #rsync安装之后要手动创建配置文件和验证密码文件

Touch/etc/rsyncd/rsyncd.secrets

chmod 600/etc/rsyncd/rsyncd.conf #密码文件的权限要设置成600, if not set to 600, the client connection will be prompted to verify that the password fails. Be sure to pay attention here. The installation is complete here. Here is the configuration.

1.2 Configuring the server side:

Vim/etc/rsyncd/rsyncd.conf #编辑rsync主配置文件

PID file =/var/run/rsyncd.pid
UID = root
GID = root
Use chroot = yes # #是否限制在指定目录, in order to install, generally need to enable
Read Only = no
Hosts allow = 10.171.22.124
Max connections = 5
Log file =/var/log/rsync.log
Transfer logging = yes
Log format =%t%a%m%f%b
Syslog facility = Local3
Timeout = 300
[Web1]
Path =/data/wwwroot
List=yes
Ignore errors
Auth users = WebUser
Secrets file =/etc/rsyncd/rsyncd.secrets
Comment = Web Home

Vim/etc/rsyncd/rsyncd.secrets #编辑密码文件

webuser:123456 #服务端的密码文件格式是用户名: Password

Join boot Start

echo "/usr/local/bin/rsync--daemon--config=/etc/rsyncd/rsyncd.conf" >>/etc/rc.local

1.3 Create a Sync account and set a password

Useradd WebUser

passwd WebUser

1.4 Starting Rsync

/usr/local/bin/rsync--daemon--config=/etc/rsyncd/rsyncd.conf

Ps-elf |grep rsync #验证是否启动成功

5 S Root 15707 1 0 0-26905 poll_s 11:06? 00:00:00 rsync--daemon--config=/etc/rsyncd/rsyncd.conf

At this point, the server configuration is over. Install the configuration client below

The traditional rsync is synchronized by crontab timing of the rsync command, so that the data can not be guaranteed real-time, so it is necessary to use inotify this software to monitor the client file changes, as long as inotify monitoring the client file changes, Then do the rsync to synchronize.

First, install rsync and Inotify-tools

Rsync is the same as the server installation, it is not written here. Install the inotify directly and configure it. Only rsync does not need to write a configuration file. Because INotify is connected to server-side synchronization by using the script to invoke the Rsync command. Since the INotify feature requires the support of the Linux kernel, it is necessary to confirm whether the Linux kernel is 2.6 or 13 before installing the Inotify-tools.

Uname-r #显示核心版本号

Ls-l/proc/sys/fs/inotify

Total 0
-rw-r--r--1 root root 0 June 12:41 max_queued_events
-rw-r--r--1 root root 0 June 12:41 Max_user_instances
-rw-r--r--1 root root 0 June 12:41 Max_user_watches

#有上面这三项输出则表示当前内核支持inotify

Tar-zxvf

wget http://sourceforge.net/projects/inotify-tools/files/inotify-tools/3.13/inotify-tools-3.13.tar.gz

TAR-ZXVF inotify-tools-3.13.tar.gz

CD inotify-tools-3.13

./configure

Make && make install

Generate Inotifywait and Inotifywatch two commands #安装完成后会在 the/usr/local/bin directory. Inotifywait can monitor any file and directory settings, and can recursively monitor the entire directory tree, Inotifywatch is used to collect statistics on the monitored file system. For example, how much inferior information each inotify event occurs.

Ls-l/usr/local/bin/inotify*

-rwxr-xr-x 1 root root 38598 Jul 2 17:48/usr/local/bin/inotifywait
-rwxr-xr-x 1 root root 40361 Jul 2 17:48/usr/local/bin/inotifywatch

Installation Complete

Second, the configuration inotify

2.1 Edit Password file

Vim/etc/rsyncd/rsync.secrets #输入服务端同步的用户密码即可.

chmod 600/etc/rsyncd/rsync.secrets #设置权限

2.2

Editing a script file

Vim/data/wwwroot/inotifyscript.sh #脚本放到要同步的目录下.

#!/bin/bash
host=10.172.186.241

src=/data/wwwroot/
Dst=web1
User=webuser

/usr/local/bin/inotifywait-mrq--timefmt '%d/%m/%y%h:%m '--format '%T%w%f%e '-e modify,delete,create,attrib $src \
| While read files
Do
/USR/BIN/RSYNC-VZRTOPG--delete--progress--password-file=/etc/rsyncd.secrets $src [email protected] $host:: $DST
echo "${files} was rsynced" >>/tmp/rsync.log 2>&1
Done

Here is a brief explanation of the script content

Host is the IP address of the server, if there is more than one, write on it in turn.

SRC is the directory that the client wants to synchronize to the server.

DFS is the WEB1 segment configured in the service-side rsyncd.conf.

User is the synchronization users configured in the server rsyncd.conf

--TIMEFMT the output format for the specified time.

--FORMAT Specifies the details of the change file.

The role of this script is to monitor the file directory changes through the inotify, found that the file changes, and then set out to synchronize the rsync operation.

2.2 Give the script executable permissions and join the boot boot

chmod +x/data/wwwroot/inotifyscript.sh

echo "/data/wwwroot/inotifyscript.sh $" >>/etc/rc.local

So far, the inotify has been configured. Synchronization verification is now possible. You can create files and directories separately, modify file contents, attribute permissions, and so on. Then observe whether the files in the server-side directory are synchronized.

This article from "Beginner's mind, always" blog, declined reprint!

Enterprise Case rsync+inotify Synchronizing Web data