Linux Operations Phase III (14) rsync

Linux operation and Maintenance Phase III (14) rsync

First, related concepts:

Rsync(remote Synchronize,rsync.samba.org): Long-distance data synchronization tool, fast synchronization of files between multiple hosts via the network, and rsync can also be used Synchronization of data in different directories on the local hard disk, using its own algorithm (rsync algorithm) to transfer only two different parts of the file , SSH mode to transfer files, so the confidentiality is good; cp,tar Compared to the backup method,rsync has the advantage of high security, fast backup and support of incremental backup , and rsync can solve the requirement of low-real-time data backup, such as regularly backing up server data to a specified server, periodically doing data mirroring on local disk, sync to other real server scenarios when Web page data changes in the load Balancer backend Real server in the cluster

with the expansion of application system scale, the security and reliability of data also put forward higher requirements,rsync in the high-end business system exposed a lot of shortcomings:rsync synchronization data, need to scan all files after the comparison, for differential transmission, If the number of files reached million or even tens of millions of levels, which will be very consumption of system performance and inefficient, sometimes change the data is often only a small part;rsync cannot monitor and synchronize data in real time, although it can trigger synchronization by scheduling tasks, but there must be a time difference between two trigger actions. This causes the server and client data to be inconsistent and can be resolved by INotify.

INotify(is a powerful, fine-grained, asynchronous FS event Monitoring mechanism, the file content change is the kernel management, the kernel will be the monitoring file itself to change the function of the output to the user space, the user space is possible to know that theLinux the kernel supports the inotify mechanism from 2.6.13, and through inotify can monitor the addition, deletion, modification, movement and other subtle events in FS, using this kernel interface, Monitor Various changes to the file under FS with third-party software (Inotify-tools)

Sersync(based on inotify development, functions like inotify,C + + writing, multithreading synchronization,Sersync You can record which files or directories are changed in the Listening directory, and then Synchronize with Rsync , only the changes will occur .

What is the difference between rsync+inotify and Rsync-sersync two architectures?

Rsync+inotify(inotify monitoring FS changes, once there are changes triggered rsync synchronization, solve the real-time data synchronization problems,inotify Only the recorded directories that have been monitored have changed, and the directories or files have not been recorded, so rsync will have to match each of the monitored directories and their files to the target directory, Then transfer the file with the difference (to traverse all the files in the directory to find out the difference in order to synchronize), if the amount of data is very large this will be very time-consuming, so inefficient)

Rsync+sersync(sersync can record which directories in the monitored directory which files have changed,rsyc sync only the changed directories or files, so fast (save time), save bandwidth , high efficiency)

Note: If the data volume is not recommendedto use rsync+inotify, if the amount of data is very large recommend the use of Rsync+sersync

Second, Operation:

1, The realization of rsync+inotify

Example: To achieve the two Web server's Web page file synchronization (one for the server, another for the client, only in the server-side update page files, view Client-side Whether there is synchronization, data flow is server-->client)

Prepare the Software:

Rsync--3.0.9.tar.gz(https://rsync.samba.org/ftp/rsync/src/rsync-3.0.0.tar.gz)

Inotify-tools-3.14.tar.gz(https://cloud.github.com/downloads/rvoicilas/inotify-tools/ inotify-tools-3.14.tar.gz)

Note:server-side installs rsync and inotify,client-side installs only rsync

#ll/proc/sys/fs/inotify/(if this directory has max_queued_events,max_user_instances,max_user_ Watches These three items the kernel supports the inotify mechanism, which is supported after the kernel 2.6.13)

Server-side(192.168.41.131):

#tar XF rsync-3.0.9.tar.gz

#cd rsync-3.0.9

#./configure--prefix=/usr/local/rsync

#make

#make Install

#vim/etc/profile.d/rsync.sh

Export path= $PATH:/usr/local/rsync/bin

#. !$

#vim/etc/man.config

Manpath/usr/local/rsync/share/man

#man rsync

#rsync Options SRC DET(rsync command format)

Options:-vzrtopg

-V (--verbose)

-Z (--compress)

-R (--recursive)

-T (--times)

-O (--owner)

-P (--perms)

-G (--group)

-A (--archive)

-A (--acls)

-H (--hard-links)

-L (--links)

-D (--devices--specials)

--delete(delete extraneousfiles from dest dirs, delete files with the target location but not the original location)

--progress(Show Progress Duringtransfer display transfer process)

--passwd-file=file(readdaemon-access password from file, specifying key file)

#echo "Redhat" >/USR/LOCAL/RSYNC/RSYNC.PASSWD(Establish authentication file, password is redhat, no user name is required here )

#chmod!$(for security, change the password file permissions to $)

#tar XF inotify-tools-3.14.tar.gz

#cd inotify-tools-3.14

#./configure--prefix=/usr/lcoal/inotify

#make

#make Install

#vim/etc/profile.d/inotify.sh

Export path= $PATH:/usr/local/inotify/bin

#. !$

#vim/etc/man.config

Manpath/usr/local/inotify/share/man

#man Inotifywatch(gather filesystem access statistics using inotifyfor short-term monitoring, task completion and results)

#man inotifywait(wait for changes-to-Files using inotifyfor continuous monitoring, real-time output results)

#inotifywait-mrq-e EVENT--timefmt fmt--format fmt FILE(inotifywait command usage)

-M (--monitor)

-R (--recursive)

-Q (--quiet)

-E (--event,Listen forspecific event (s) only)

--timefmt

--format(Output in a user-specified format)

The EVENT is:

Modify(a watched file or afile within a watched directory was written to)

Delete(a file or directorywithin a watched directory was deleted)

Create(a file or Directorywas created within a watched directory)

attrib(the metadata of awatched file or a file within a watched directory was modified)

Move(A file or Directorywas moved from or to A watched directory)

#vim/root/rsync.sh

#!/bin/bash

#

host=192.168.41.132

src=/var/www/html/

Des=test

User=testuser

/usr/local/inotify/bin/inotifywait-mrq--timefmt '%d/%m/%y%h:%m '--format '%T%w%f%e '-e Modify,delete,create,attri  b $src | While Read Files;do

/USR/BIN/RSYNC-VZRTOPG--delete--progress--passwd-file=/usr/local/rsync/rsync.passwd $src [email protected] $host:: $des

echo "${files} was rsynced" >>/var/log/rsync.log 2>&1

Done

#chmod 764/root/rsync.sh

#sh/root/rsync.sh &(note!) Wait for Client-side to start the Rsync service before running this script)

Client-side(192.168.41.132):

#tar XF rsync-3.0.9.tar.gz

#cd rsync-3.0.9

#./configure--prefix=/usr/local/rsync

#make

#make Install

#vim/etc/profile.d/rsync.sh

Export path= $PATH:/usr/local/rsync/bin

#. !$

#vim/etc/man.config

Manpath/usr/local/rsync/share/man

#echo "Testuser:redhat" >/USR/LOCAL/RSYNC/RSYNC.PASSWD(Establish certification document )

#chmod!$(for security, change the password file permissions to $)

#vim/usr/local/rsync/rsync.conf

UID = root

GID = root

Use chroot = no(default to trueto increase backup of soft links to directory files)

Max connections = 10

Timeout = 600

Strict modes = yes

PID file =/var/run/rsyncd.pid

Lock file =/var/log/rsyncd.lock

Log file =/var/log/rsyncd.log

MOTD file =/etc/rsyncd.motd(You can edit the welcome message at startup in this file)

[Test] (custom name)

Path =/var/www/html/(data synchronization to local directory path)

Comment = Test(description information)

Ignore errors = yes

Read Only = no(set the rsync server file as read-write permission)

Hosts allow = 192.168.41.131(host address that allows data synchronization, multiple separated by commas)

Hosts deny = *(host address that disables data synchronization)

List = False(does not display the rsync server-side resource listing)

UID = root(set rsync run permissions)

GID = root

Port = 873(default port)

Auth users = testuser(user name to perform data synchronization, you can set multiple comma-separated)

Secrets file =/user/local/rsync/rsync.passwd(user authentication profile with user name and password)

#rsync--daemon--config=/usr/local/rsync/rsync.conf(start rsync)

Test:

Server-side (/var/www/html ):

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M02/77/10/wKiom1ZiMsnS4l-qAABrmV9sixw732.jpg "title=" Iii1401.jpg "alt=" Wkiom1zimsns4l-qaabrmv9sixw732.jpg "/>

Client-side (/var/www/html ):

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M02/77/0F/wKioL1ZiM0XgifzPAAAb2Kb_Lvo556.jpg "title=" Iii1402.jpg "alt=" Wkiol1zim0xgifzpaaab2kb_lvo556.jpg "/>

2, The realization of Rsync+sersync:

Server-side:

For example, install rsync, make an authentication file, and change the permissions to 600

#mkdir/usr/local/sersync/

# TAR-XF Sersync_64bit_binary_stable_final.tar.gz

#cd gnu-linux-x86

#mv Confxml.xml sersync2/usr/local/sersync/

#cd/usr/local/sersync

#cp Confxml.xml Confxml.xml.bak

#vim Confxml.xml

......

<sersync>

<localpath watch= "/var/www/html" >

<remote ip= "192.168.41.132" name= "test"/>

<!--<remote ip= "192.168.8.39" name= "Tongbu"/>-->

<!--<remote ip= "192.168.8.40" name= "Tongbu"/>-->

</localpath>

<rsync>

<commonparams params= "-vzrtopg--delete--progress"/>

<auth start= "true" users= "testuser" passwordfile= "/usr/local/rsync/rsync.passwd"/>

<userdefinedport start= "false" port= "874"/><!--port=874--

<timeout start= "false" Time= "" "/><!--timeout=100--

<ssh start= "false"/>

</rsync>

<faillog path= "/tmp/rsync_fail_log.sh" timetoexecute= "/><!--default every 60mins execute once-->

<crontab start= "true" schedule= "><!--600mins-->

<crontabfilter start= "false" >

<exclude expression= "*.php" ></exclude>

<exclude expression= "info/*" ></exclude>

</crontabfilter>

</crontab>

<plugin start= "false" name= "command"/>

</sersync>

......

#vim/root/check_sersync.sh

#!/bin/bash

#

Sersync= "/usr/local/sersync/sersync2"

Conf_file= "/usr/local/sersync/confxml.xml"

status= ' ps aux | grep ' Sersync2 ' | Grep-v ' grep ' | Wc-l '

If [$STATUS-eq 0];then

$SERSYNC-D-r-o $CONF _file &

Else

Exit 0;

Fi

#crontab-E ( This script is detected every 5 minutesto prevent sersync2 accidental interruption , and if Sersync2 is not running, it will be automatically turned on for real-time synchronization)

*/5 * * * * bash/root/check_sersync.sh&>/dev/null

#/usr/local/sersync/sersync2-d-r-o/usr/local/sersync/confxml.xml(open sersync service)

-D (run as a daemon)

-R (rsync all the localfiles to the remote servers before the Sersync work)

-O (config XML name: ./confxml.xml)

Client-side:

refer to the example above, install rsync, configure the rsync.conf file, configure the authentication file, and turn on the rsync service

Test:

Server-side:

650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M02/77/10/wKiom1ZiMvyAkYQbAABeKZMPEE4022.jpg "title=" Iii1403.jpg "alt=" Wkiom1zimvyakyqbaabekzmpee4022.jpg "/>

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M01/77/0F/wKioL1ZiM3iA804-AAA3yYJJMzY516.jpg "title=" Iii1404.jpg "alt=" Wkiol1zim3ia804-aaa3yyjjmzy516.jpg "/>

Client-side:

650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M00/77/10/wKiom1ZiMyGSuvUTAAAqecFM-T8276.jpg "title=" Iii1405.jpg "alt=" Wkiom1zimygsuvutaaaqecfm-t8276.jpg "/>

The above content is to collect data.

This article is from the "Linux operation and maintenance of difficult learning notes" blog, declined reprint!

Linux Operations Phase III (14) rsync