Linux implementation Inotify+rsync Real-time synchronization system

1, testing the distributor system to meet the requirements, the kernel is more than 2.3

Uname-r #查看系统内核 ll/proc/sys/fs/inotify/

　　

Meet the requirements.

2. Download the latest version and compile the installation on the Distributor server

CD  #回到用户目录wget http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz  #下载最新版tar ZXVF inotify-tools-3.14.tar.gz  #解压到当前目录cd inotify-tools-3.14 #进入目录./configure--prefix=/alidata/server/inotify /#编译配置, here is the specified location (your file directory) make #编译make install #安装make clean #从源文件夹清除二进制对象等 cd/alidata/server/inotify/bin./inotifywait-- Help  #查看是否安装完成一般有错误上面几个步骤会有提示
./inotifywait-mrq--format '%xe%w%f '-e modify,create,delete,attrib/alidata/tmp/#m是开启监测, R-Recursive directory,/alidata/ Directory for TMP monitoring Touch/alidata/tmp/test1 #新开个tty创建个文件会有如输出

#为方便使用可以选择设置 # Set System environment variable, add soft connection #echo "path= $PATH:/alidata/server/inotify/bin" >>/etc/profile.d/inotify.sh # source/etc/profile.d/inotify.sh #使设置立即生效 #echo "/alidata/server/inotify/lib" >/etc/ld.so.conf.d/inotify.conf && ldconfig#ln-s/alidata/server/inotify/include/usr/include/inotify

　　

At this point, the distribution server file changes can be detected.

3. Compile and install rsync to the content distributor here to act as the client's role

wget Https://download.samba.org/pub/rsync/src/rstar zxvf rsync-3.1.2.tar.gzcd rsync-3.1.2./configure--prefix=  /alidata/server/rsyncmakemake Install

Next is the file synchronization, the file changes detected after the active push to the target server, the details said later.

4. Compile and install rsync to the target server and enable the service

wget Https://download.samba.org/pub/rsync/src/rstar zxvf rsync-3.1.2.tar.gzcd rsync-3.1.2./configure--prefix=  /alidata/server/rsyncmakemake Install

To add a master configuration file:

Nano/etc/rsyncd.conf

Copy the following code

UID = Rootgid = Rootport = 873use Chroot = nohosts allow = *max connections = 3MOTD file =/alidata/passwd/rsync/rsyncd.mo T #问候语pid file =/alidata/server/rsync/rsyncd.pidlock File =/alidata/server/rsync/rsync.locklog File =/alidata/log/ Rsync/rsyncd.logtransfer logging = yes #传输日志ignore errors #忽略错误 [Netho]path =/alidata/tmp/rsynctest #同步路径auth users = Neth o #使用用户必须是服务器真实用户secrets file =/alidata/passwd/rsync/rsyncd.pass #用户认证文件list = Noread only = no #只读

Establish the authentication file, I use the Apache Authentication tool to generate a more complex password (too troublesome to manually enter the Echo Woshimima >/alidata/passwd/rsync/rsyncd.pass)

Mkdir-p/alidata/passwd/rsycn htpasswd-cb/alidata/passwd/rsync/rsyncd.pass Netho Netho123456789
chmod 600/alidata/passwd/rsync/rsyncd.pass    #为什么要这一步呢? and strict modes =yes/no have a relationship, no time to experiment, rub

Set RSYNCD.MOTD file;

Nano/alidata/passwd/rsync/rsyncd.mot

Copy the following code

++++++++++++++++++++++++++++++++++++++++++++++ Welcome to use the  rsync services!           2002------2009 ++++++++++++++++++++++++++++++++++++++++++++++

To start the Rsync service:

Cd/alidata/server/rsync/bin./rsync  --daemon--config=/etc/rsyncd.conf &

Open the specified port

UFW Allow 873/tcp

Test links

Ps-ef |grep rsync #进程netstat-anop | grep rsync #端口信息./rsync-rdt rsync://127.0.0.1:873/#看到我们设置的欢迎信息 (mot) Description startup successful service OK

　　

How to stop??? I don't know, ^_^.

To facilitate our use of scripting to manage their inclusion in the service

Touch/etc/init.d/rsyncd
chmod +X/ETC/INIT.D/RSYNCD
Nano/etc/init.d/rsyncd

Copy the following code

#!/bin/bash # Description:rsync server# processname:rsyncd status1=$ (ps-ef | egrep "Rsync--daemon.*rsyncd.conf" | grep -V ' grep ') pidfile= "/alidata/server/rsync/rsyncd.pid" rsync= "/alidata/server/rsync/bin/rsync" start_rsync= "${rsync             }--daemon--config=/etc/rsyncd.conf "function Rsyncstart () {if [" ${status1}x "= =" X "];then ${start_rsync} status2=$ (Ps-ef | egrep "Rsync--daemon.*rsyncd.conf" | grep-v ' grep ') if ["${status2}x" ! = "X"];then echo "rsync service start ....         Ok "fi else echo" rsync service is running! " fi} function Rsyncstop () {if ["${status1}x"! = "X"];then kill-9 $ (cat $pidfile) rm-f $PIDF Ile status2=$ (Ps-ef | egrep "Rsync--daemon.*rsyncd.conf" | grep-v ' grep ') if ["${statusw2}x" = = "X"] And then echo "rsync service stop .... Ok "fi else echo" rsync serviceis not running! "            fi} function Rsyncstatus () {if ["${status1}x"! = "X"];then echo "rsync service is running!"       else echo "Rsync service is not running!"            fi} function Rsyncrestart () {if ["${status1}x" = = "X"];then echo "rsync service is not running ..." Rsyncstart else rsyncstop rsync--daemon--config=/usr/local/server/rsync/etc/rsyncd.co  NF status2=$ (Ps-ef | egrep "Rsync--daemon.*rsyncd.conf" | grep-v ' grep ') if ["${status2}x" ! = "X"];then echo "rsync service start ....          OK "Else echo" rsync service is not running! "          Fi fi} case "$" in "start") Rsyncstart;;          "Stop") rsyncstop;;          "Status") rsyncstatus;; "Restart") Rsyncrestart               ;; *) echo Echo "Usage: $ start|stop|restart|status" echo Esacexit 0

Backup

#! /bin/sh### BEGIN INIT info# provides:rsyncd# required-start: $remote _fs $syslog # required-stop: $remote _f S $syslog # Should-start: $named # default-start:2 3 4 * default-stop:# short-description:fast remote file copy Program daemon# Description:rsync are a program this allows files to being copied to and# from Remot  E machines in much the same is rcp.# this provides RSYNCD daemon functionality.### END INIT infoset -e#/etc/init.d/rsync:start and stop the rsync daemondaemon=/usr/bin/rsyncrsync_enable=falsersync_opts= ' RSYNC_ Defaults_file=/etc/default/rsyncrsync_config_file=/etc/rsyncd.confrsync_pid_file=/var/run/rsync.pidrsync_nice_ Parm= ' rsync_ionice_parm= ' test-x $DAEMON | | Exit 0. /lib/lsb/init-functionsif [-S $RSYNC _defaults_file]; Then.        $RSYNC _defaults_file Case "x$rsync_enable" in xtrue|xfalse);;        XINETD) exit 0;; *) log_failure_mSG "Value of rsync_enable in $RSYNC _defaults_file must is either ' true ' or ' false ';"                        Log_failure_msg "not starting rsync daemon."    Exit 1;;        ESAC case "X$rsync_nice" in x[0-9]) rsync_nice_parm= "--nicelevel $RSYNC _nice";;        X1[0-9]) rsync_nice_parm= "--nicelevel $RSYNC _nice";;        x);; *) log_warning_msg "value of Rsync_nice in $RSYNC _defaults_file must is a Value between 0 and (inclusive);    "Log_warning_msg" ignoring rsync_nice now.;        ESAC case "X$rsync_ionice" in x-c[123]*) rsync_ionice_parm= "$RSYNC _ionice";;        x);;                        *) log_warning_msg "Value of Rsync_ionice in $RSYNC _defaults_file must be-c1,-c2 or-c3;"                        Log_warning_msg "Ignoring rsync_ionice now."    ;; Esacfiexport path= "${path:+ $PATH:}/usr/sbin:/sbin "Rsync_start () {if [!-S" $RSYNC _config_file "]; then log_failure_msg" Missing or empty CONFIG FILE $RSYN C_config_file "Log_end_msg 1 Exit 0 fi # See Ionice (1) If [-N" $RSYNC _ionice_parm "] && [ -x/usr/bin/ionice] &&/usr/bin/ionice "$RSYNC _ionice_parm" true 2>/dev/null; Then/usr/bin/ionice "$RSYNC _ionice_parm"-p$$ >/dev/null 2>&1 fi if Start-stop-daemon--start--q Uiet--background--pidfile $RSYNC _pid_file--make-pidfile $RSYNC _nice_parm--exec $DAEMON----no -detach--daemon--config "$RSYNC _config_file" $RSYNC _opts then rc=0 sleep 1 if! Kill-0 $ (Cat $RSYNC _pid_file) >/dev/null 2>&1;  Then log_failure_msg "Rsync Daemon failed to start" rc=1 fi else rc=1 fi if [ $RC-eq 0]; Then log_end_msg 0 Else log_end_msg 1 rm-f $RSYNC _pid_file fi} # rsync_startcase "$" In start) If "$RSYNC _enable"; Then log_daemon_msg "Starting rsync daemon" "rsync" if [-S $RSYNC _pid_file] && kill-0 $ (c At $RSYNC _pid_file) >/dev/null 2>&1;            Then log_progress_msg "apparently already running" Log_end_msg 0 exit 0 Fi Rsync_start Else if [-s] $RSYNC _config_file "];  Then ["$VERBOSE"! = no] && log_warning_msg "rsync daemon not enabled in $RSYNC _defaults_file, not  Starting ... "fi fi;; Stop) log_daemon_msg "stopping rsync daemon" "rsync" Start-stop-daemon--stop--quiet--oknodo--pidfile $RS        Ync_pid_file log_end_msg $?  Rm-f $RSYNC _pid_file;; Reload|force-reload) log_warning_msg "reloading rsync daemon:not needed, as the daemon" log_warning_msg "re  -reads the config file whenever a client connects. "; Restart) Set +e if $RSYNC _enable; Then log_daemon_msg "Restarting Rsync daemon" "rsync" if [-S $RSYNC _pid_file] && kill-0 $ (Cat $RSYNC _pid_file) >/dev/null 2>&1; Then Start-stop-daemon--stop--quiet--oknodo--pidfile $RSYNC _pid_file | | True sleep 1 Else log_warning_msg "rsync Daemon not running, attempting to start. "Rm-f $RSYNC _pid_file fi rsync_start else if [-S" $RSYNC _config_ FILE "];  Then ["$VERBOSE"! = no] && log_warning_msg "rsync daemon not enabled in $RSYNC _defaults_file, not  Starting ... "fi fi;; Status) status_of_proc-p $RSYNC _pid_file "$DAEMON" RSYNC exit $?  # notreached due to set-e;; *) echo "Usage:/etc/init.d/rsync {start|stop|reload|force-reload|restart|status}" exit 1esacexit 0

　　Set boot up

echo "/ETC/INIT.D/RSYNCD start" >>/etc/rc.local

5. After the target server is well-prepared, write the monitor file changes on the content distributor to initiate synchronization scripts rstnc-inotify.sh

#!/bin/bashserver=/alidata/server/rsync=${server}rsync/bin/rsyncinotify=${server}inotify/binsrc=/alidata/tmp/ mubiao/
log=/alidata/log/rsync-inotify.log# the source path that needs to be synchronized des=netho# the name in [] in the configuration on the target server. rsync_passwd_file=/alidata/passwd/rsync/rsyncd.pass# rsync verified password file ip=127.0.0.1# target server 1ip1=192.168.0.19# target server 2user =netho# rsync--daemon defined authentication User name CD ${src}# This method, due to the characteristics of rsync synchronization, here must first CD to the source directory, # INotify again listen./To Rsync synchronization after the directory structure is consistent, ${inotify}/ Inotifywait-mrq--format '%xe%w%f '-e modify,create,delete,attrib,close_write,move./| While read file# monitors the list of file paths that have changed loop do ino_event=$ (echo $file | awk ' {print $} ') # assigns the inotify output cut to the event type portion to Ino_ EVENT ino_file=$ (echo $file | awk ' {print $} ') # Assign the inotify output to Ino_file echo "in the File path section--------------- ----------------$ (date)------------------------------------">> ${log} 2>&1 echo $file >> ${log}        2>&1 # Add, modify, write complete, move into event # Add, change and put in the same judgment, because they are certainly for the file operation, even if the new directory, to synchronize is only an empty directory, does not affect the speed. if [[$INO _event =~ ' CREATE '] | | [[$INO _event =~ ' MODIFY '] | | [[$INO _event =~ ' close_write '] | | [[$INO _event =~ ' moved_to ']] # Judging event type then echo ' CREATE or MODIFY or close_write or moved_to ' >> ${log} 2>& 1 # ${rsync}-avzcr--port 3375--password-file=${rsync_passwd_file} $ (dirname ${ino_file}) ${user}@${ip1} :: ${des} && # Ino_file variable represents path oh-C checksum file contents ${rsync}-AVZCR--port 3375--password-file= ${rsync_passwd_file} $ (dirname ${ino_file}) ${user}@${ip}::${des} # Take a closer look at the rsync Sync command source above is a $ (dirname ${ino_fi LE}) Liang # that is, each time only the targeted synchronization of the file directory changes (the method of synchronizing the target files in some extreme environment of the production environment will drain the file #现在可以在不漏文件下也有不错的速度 to achieve balance) and then use the-R The parameter recursively passes the directory structure of the source to the target to ensure the consistency of the directory structure fi #删除, move out event if [[$INO _event =~ ' DELETE '] | |                [[$INO _event =~ ' moved_from '] then echo ' DELETE or Moved_from ' >> ${log} 2>&1 #${rsync}-AVZR--delete--password-file=${rsync_passwd_file} $ (dirname ${ino_file}) ${user}@${ip1}::${des} & & ${rsync}-AVZR--delete--porT 3375--password-file=${rsync_passwd_file} $ (dirname ${ino_file}) ${user}@${ip}::${des} # See the rsync command if the direct sync is already Deleted path ${ino_file} will be reported no such or directory error so here the source of synchronization is deleted file or directory of the previous level path, # and add--delete to delete the target has not the source of the file, here can not do the specified file deletion, if the path to delete the more Close to the root, the synchronized directory is more than a month, and the operation to delete synchronously takes more time.        There are better ways to students, welcome to exchange. Fi #修改属性事件 refers to touch chgrp chmod Chown and other operations if [[$INO _event =~ ' ATTRIB ']] then echo ' ATT                RIB ' IF [!-d ' $INO _file "] # If you modify the properties of the directory is not synchronized, because the synchronization directory will have a recursive scan, and so on when the files in this directory synchronization, rsync will update this directory in passing. Then # ${rsync}-AVZCR--password-file=${rsync_passwd_file} $ (dirname ${ino_file}) $ {user}@${ip1}::${des} && ${rsync}-avzcr--port 3375--password-file=${rsync_passwd_file} $ (DirName ${ino_file}) ${user}@${ip}::${des} fi fidone

Reference http://www.ttlsa.com/web/let-infotify-rsync-fast/

chmod +x./rsync-inotify.sh #赋予执行权限./rsync-inotify.sh & #后台执行
echo "/alidata/netho/rsync-inotify.sh &" >>/etc/rc.local

To set a timing scheme to prevent script problems

　　

Report:

The command format for rsync can be in the following six ways:

rsync [OPTION] ... SRC DEST

rsync [OPTION] ... SRC [[Email protected]]host:dest

rsync [OPTION] ... [[email protected]] HOST:SRC DEST

rsync [OPTION] ... [[email protected]] HOST::SRC DEST

rsync [OPTION] ... SRC [[Email protected]]host::D EST

rsync [OPTION] ... rsync://[[email protected]]host[:P ort]/src [DEST]

There are six different modes of operation for Rsync, which corresponds to the above six command formats:

1) Copy the local file. This mode of operation is initiated when both the SRC and DES path information do not contain a single colon ":" delimiter.

2) Use a remote shell program (such as rsh, SSH) to copy the contents of the local machine to the remote machine. This mode is started when the DST path address contains a single colon ":" delimiter.

3) Use a remote shell program (such as rsh, SSH) to copy the contents of the remote machine to the local machine. This mode is started when the SRC address path contains a single colon ":" delimiter.

4) Copy files from the remote rsync server to the local machine. This mode is started when the SRC path information contains the "::" delimiter.

5) Copy files from the local machine to the remote rsync server. This mode is started when the DST path information contains the "::" delimiter.

6) The list of files for the remote machine is listed. This is similar to the rsync transfer, but only if the local machine information is omitted from the command.

Parameter description

-V,--verbose verbose mode output
-Q,--quiet thin output mode
-C,--checksum turn on the check switch to force verification of file transfers
-A,--archive archive mode, which means to transfer files recursively and keep all file attributes equal to-rlptgod
-R,--recursive subdirectories in recursive mode
-R,--relative using relative path information
-B,--backup creates a backup, that is, the old file is renamed to ~filename when the same file name exists for the purpose. You can use the--suffix option to specify a different backup file prefix.
--backup-dir store backup files (such as ~filename) in the directory.
-suffix=suffix Defining backup File prefixes
-U,--update only updates, which is to skip all the files that already exist in DST, and the file time is later than the file to be backed up. (Does not overwrite the updated file)
-L,--links reserved Soft link
-L,--copy-links to handle soft links like regular files
--copy-unsafe-links only copies links to links outside the SRC Path directory tree
--safe-links ignoring links to the SRC Path directory tree
-H,--hard-links reserved hard link-p,--perms keep file permissions
-O,--owner hold the file belongs to the main information-G,--group keep the file group information
-D,--devices keep device file information-T,--times keep file time information
-S,--sparse special processing of sparse files to save DST space
-N,--dry-run reality which files will be transmitted
-W,--whole-file copy files without incremental detection
-X,--one-file-system do not cross file system boundaries
-B, the block size used by the--block-size=size test algorithm, is 700 bytes by default
-E,--rsh=command specifies data synchronization using RSH and SSH
--rsync-path=path Specify the path information for the rsync command on the remote server
-C,--cvs-exclude uses the same method as CVs to automatically ignore files to exclude files that you do not want to transfer
--existing only updates those files that already exist in DST without backing up those newly created files
--delete Delete those files that are not in the DST SRC
--delete-excluded also deletes those files that are excluded by the option specified by the Receive side
--delete-after transfer ends after removal
--ignore-errors Timely IO errors are also deleted
--max-delete=num deleting NUM files up to a maximum
--partial retains files that are not fully transmitted for any reason, to expedite subsequent transmissions
--force forcibly delete a directory, even if it is not empty
--numeric-ids does not match the user and group ID of a number to a user name and group name
--timeout=time IP time-out, in seconds
-I,--ignore-times do not skip files that have the same time and length
--size-only when deciding whether to back up a file, just look at the file size regardless of file time
--modify-window=num determines whether the file is time-stamped with the time Stamp window, which defaults to 0
-t--temp-dir=dir creating temporary files in Dir
--compare-dest=dir also compares the files in DIR to determine if a backup is required
-p equivalent to--partial
--progress Show Backup process
-Z,--compress compress the backed-up files during transmission
--exclude=pattern specify to exclude file modes that do not need to be transferred
--include=pattern specifies file modes that need to be transferred without exclusion
--exclude-from=file exclude files in the specified schema in file
--include-from=file does not exclude files that specify pattern matching
--version Print version Information
--address binding to a specific address
--config=file specify a different configuration file, do not use the default rsyncd.conf file
--port=port Specify a different rsync service port
--blocking-io using blocking IO for remote shells
-stats gives the transfer status of some files
--progress in the transmission of the real-time transmission process
--log-format=format specifying the log file format
--password-file=file get the password from file
--bwlimit=kbps limit I/O bandwidth, Kbytes per second-h,--help display Help information

Rsync Client Configuration

Linux implementation Inotify+rsync Real-time synchronization system