Real-time synchronous transmission of data based on Rsync+inotify

Objective

Compared with the traditional methods of CP and Tar Backup, rsync has the advantages of high security, fast backup, support of incremental backup, and so on, while rsync can solve the requirement of low-real-time data backup, but with the increasing number of files and the requirement of real-time synchronization, Rsync can't meet the demand. +inotify was born. This article will explain the basic knowledge of rsync and how to realize real-time synchronous transmission of data based on Rsync+inotify.

Rsync Related Introduction

rsync (remote sync) is a fast incremental backup tool that supports local replication or synchronizes with other SSH (secure transfers) and rsync hosts.

Characteristics

① can be mirrored to save the entire directory tree or file system

② Higher data transfer efficiency

③ enables secure data transfer with SSH

④ supports anonymous transfers

Working mode

①shell mode, also known as local mode

② Remote Shell mode, you can use the SSH protocol to host its remote transfer process

③ list mode, listing only the content in the source,-NV

④ service mode, at which time Rsync works as a daemon and can receive data synchronization requests from clients

Command options

-N: Synchronous test, does not perform the true synchronization process;-V: Verbose output mode-Q: Silent mode-c:checksum, turn on check function-r: Recursive copy-A: archive, retain the original properties of the file;-P: Retain the file's permissions;-T: Preserves the file's timestamp; : Reserved group-o: Reserved master-d: Reserved device file-e ssh: use SSH as transport bearer;--progress: Show progress bar--stats: show how to perform compression and transfer

Note: In the rsync command, if the source path is a directory and there is/at the end of the copy path, the contents of the directory will be copied, not the directory itself, and if not at the end, all files in the directory itself and in the directory will be synchronized, or not at the end of the target path, such as: Rsync-r/var/log/ /tmp #复制 All files in/var/log directory, excluding log itself rsync-r/var/log/tmp #复制/var/log Entire Directory

rsync synchronization data, need to complete scan files, compared to the changes in the file, if the number of files is too large, so scanning is very time-consuming, and rsync can be achieved by crontab regular synchronization, but also not up to the requirements of real-time synchronization, to solve this shortcoming, you need inotify.

Real-time data synchronization implementation

inotify Related Introduction

Can be used to define all the files under the specified directory, and once the metadata of the files changes, the rsync is notified to synchronize the data to achieve real-time synchronization

Deployment process

Let's take a look at how rsync+inotify achieves data synchronization

Case requirements

Web server 1:172.16.10.100 (master server)

Web server 2:172.16.10.212 (secondary server)

Two Web servers that require data to be updated on the primary server and the secondary server to update the data automatically

rsync role Description: Because inotify is monitoring the client in rsync, and is active synchronization, so in this scenario, the master server is the Rsync client, the secondary server is the rsync server side

Rsync Server-side configuration

[[email protected] ~]# yum install xinetd -y   #安装守护进程, the rsync system comes with it, [[email protected] ~]# vim /etc/xinetd.d/rsync       if not installed     #修改此项         disable = no  #启用rsync [[ email protected] ~]# vim /etc/rsyncd.conf  #为rsync创建配置文件, this file is not # global  by default settingsuid = nobody                        #运行rsync用户gid  = nobody                       # Run the rsync group use chroot = no                     #关闭chrootmax  connections = 10                #最大并发连接数strict  modes = yes                  #开启严格模式pid  file = /var/run/ rsyncd.pid     #pid文件位置log  file = /var/log/rsyncd.log      #日志文件位置 # directory to be synced[web]                                #文件共享名path  = /web                         #共享文件路径ignore  errors =  yes                #忽略错误read  only = no                      #不允许读write  only = no                     #不允许写hosts  allow = 172.16.0.0/16         #白名单hosts  deny = *                      #黑名单list  = false                         #不允许列出文件uid  = root                          #共享已root用户运行, can override global configuration gid  = root                          #共享已root组运行, can override global configuration Auth users = scholar                #用户认证secrets  file = /etc/rsync.passwd    #认证文件 [[email protected] ~]# vim /etc/rsync.passwd  #创建认证文件 # format: Username:password, This file cannot allow other users to have access, and the password cannot exceed 8 characters scholar:scholar[[email protected] ~]# chmod 600 /etc/ rsync.passwd  #设置权限

Start the service, set the boot up, and view the listening port

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/07/wKiom1U936bxeqjyAAB89RSJ0Gc120.jpg "title=" 1.jpg " alt= "Wkiom1u936bxeqjyaab89rsj0gc120.jpg"/>

Rsync Client Configuration

Create an authentication file with just a password

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/03/wKioL1U95iDiItirAABzsOuPt0w948.jpg "title=" 2.jpg " alt= "Wkiol1u95idiitiraabzsoupt0w948.jpg"/>

Methods for testing data synchronization

Access via rsync daemon:   pull: rsync [option ...]  [[email protected]]HOST::SRC... [DEST]    such as: rsync --password-file=/etc/ rsync.passwd [email protected]::web /web       rsync [ OPTION ...]  rsync://[[email protected]]host[:P ort]/src... [dest]    such as: rsync -- password-file=/etc/rsync.passwd rsync://[email protected]/web /web   push:  Rsync [option ...]  src... [[email protected]]host::D est    such as: rsync --password-file=/etc/ rsync.passwd /web [email protected]::web          Rsync [option ...]  src... rsync://[[email protected]]host[:P ort]/dest    such as: rsync -- Password-file=/etc/rsync.passwd /web rsync://[email protected]/web

installation Inotify-tools

[Email protected] ~]# tar xf inotify-tools-3.14.tar.gz [[email protected] ~]# CD Inotify-tools-3.14[[email protected] Ino tify-tools-3.14]#./configure[[email protected] inotify-tools-3.14]# make && make install

Create an rsync script

[Email protected] ~]# vim/usr/local/bin/rsyncd#!/bin/bashsrc=/web/dest=webhost=172.16.10.212/usr/local/bin/ Inotifywait-mrq--timefmt '%d/%m/%y%h:%m '--format '%T%w%f%e '-E modify,delete,create,attrib $SRC | While read Files;do rsync-vzrtopg--delete--progress--password-file=/etc/rsync.passwd $SRC [email Protec Ted] $HOST:: $DESTdone #inotifywait parameter in detail: #-m, indicating that the event listening state is always maintained #-r, indicating a recursive query directory #-q, indicating that the print out monitoring event #-e, specify the events to monitor, including modify, delete, Create, attrib, etc. #--timefmt: Specify the time output format #--format: Specify the details of the change file

Set up auto run on boot

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/6C/09/wKiom1U-AbHQezqmAABhBvMkLEY927.jpg "title=" 3.jpg " alt= "Wkiom1u-abhqezqmaabhbvmkley927.jpg"/>

Test synchronization

We add a site on the main server, the site is ready for us to copy directly into

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/6C/0C/wKiom1U-FBrBS07CAABfl7NdwSM778.jpg "title=" 4.jpg " alt= "Wkiom1u-fbrbs07caabfl7ndwsm778.jpg"/>

Let's go to the auxiliary server and see if the data is synchronized.

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/08/wKioL1U-FgTiOEQGAAJsKvu4t2Q984.jpg "title=" 5.jpg " alt= "Wkiol1u-fgtioeqgaajskvu4t2q984.jpg"/>

Synchronization is successful, so the data real-time synchronization configuration is completed based on Rsync + inotify

The end

Well, rsync + inotify here, if you want to implement multiple from the server synchronization, more than a few rsync server side is good, the corresponding master server INotify script also need to be slightly modified, the process is not troublesome, the experiment process encountered problems can leave a message. The above is only for individual learning to organize, if there are mistakes, big God do not spray ~ ~ ~

This article is from the "North Scholar" blog, please make sure to keep this source http://scholar.blog.51cto.com/9985645/1639713

Real-time synchronous transmission of data based on Rsync+inotify