Use unison + INotify + rsync to implement file-triggered real-time one-way/bidirectional/multidirectional file synchronization

The project has a new demand, in more than one app server randomly upload Pictures/resource files, one of the app needs to read the uploaded images, so that the previous rsync+inotify settings will not be able to deal with.

So we study the next multi-directional trigger real-time synchronization mechanism, we can experiment, if there is inappropriate place, please contact me to amend, thank you.

Here's the start.

First, Introduction

Unison is a file Sync tool that can be used under Windows, Linux, and other Unix platforms to keep content consistent between two folders (local or network). Unison has the same features as some other synchronization tools or file systems, but it also has its own characteristics:

1. Cross-platform use;

2. No special requirements for kernel and user rights;

3.Unison is bidirectional, it can automatically handle the two copy of the update without conflicting parts, the conflicting parts will be displayed to allow users to choose the update strategy;

4. As long as the two host can be connected, you can run the unison, you can directly use the socket connection or secure SSH connection, the bandwidth requirements are not high, using similar rsync compression Transfer Protocol

5.Unison Bidirectional Synchronization Basic principle is: if there is a B two folders, a folder to sync their own changes to the B,b folder also to sync their own changes to a, and finally a B two folders of the same content, is the AB folder collection.

6.Unison bidirectional synchronization does not apply to the same file in two synchronized folders, at the same time being modified, Unison will not be synchronized, because Unison can not judge that whichever. Unison creates a file lock and is manually unlocked by the user.

Second, the test environment

A-machine Ubuntu 12.04 10.8.12.35 ssh:22

B Machine RHEL6.2 10.8.12.201 ssh:22

III. Deployment Results

Realize bidirectional synchronization of the/home/test/reward/directory of A/b machine.

Iv. test Results

A) A/b machine corresponding to the directory to achieve two-way synchronization, based on the file level, there are authentication measures, there is a log, if the transmission fails, will be transmitted again.

b) There is a emptying protection mechanism: if there is a rm-rf./* operation will automatically stop syncing, and then the log error. But at this point there is a change in the file level, perform the empty sync operation, and then synchronize the changed files

c) log file/home/test/inotify_unison/unison.log can not be deleted, once deleted, unison program error.

D) Note that the operation of the file is carried out on that machine, and the log will be hit on that machine.

Five, configuration steps

A) Install the software

A machine, Ubuntu system

[email protected]:~$ sudo apt-get install ocaml unison inotify-tools – y[email protected]:~$ dpkg -l | grep ocamlii  ocaml                               3.12.1-2ubuntu2                ML language implementation with a  class-based object systemii  ocaml-base                          3.12.1-2ubuntu2                runtime system for ocaml bytecode executablesii  ocaml-base-nox                     3.12.1-2ubuntu2                Runtime system for OCaml  bytecode executables  (no x) ii  ocaml-interp                       3.12.1-2ubuntu2                OCaml  interactive interpreter and standard librariesii  ocaml-nox                           3.12.1-2ubuntu2                ML implementation with a class-based object system  (no x) [EMAIL PROTECTED]:~$&NBsp;dpkg -l | grep unisonii  unison                              2.40.65-1ubuntu1               file-synchronization tool for unix and windows[email protected]:~$  dpkg -l | grep inotify-toolsii  inotify-tools                      3.13-3                          command-line programs providing a simple  Interface to inotify

B-Machine Rhel system

[email protected] ~]# Yum install unison240.x86_64 inotify-tools-yinstalled:unison240-gtk.x86_64 0:2.40.102-5.                                                                                          El6inotify-tools.x86_64 0:3.14-1.el6 Dependency Installed:unison240.x86_64 0:2.40.102-5.el6

b) Configure a B/C machine SSH trust account to test

Slightly

c) Check the machine time synchronization mechanism to ensure that the AB machine time is consistent.

Slightly

d) Configuring a Unison service on a machine

1) Execute the Unison command to generate the configuration file

[Email protected]:~$ Unison ABC

2) Create a log folder (this step can be ignored and added to the script)

[Email protected]:~$ Mkdir/home/test/inotify_unison

3) Configure Unison service

[Email protected]:~$ vim/home/test/.unison/default.prf

# Unison Preferences File

#表示全自动模式, accept and execute the default action.

Batch = True

#表示保持同步的文件属组信息.

Group = True

#表示保持同步的文件属主信息.

Owner = True

#同步指定的子目录及文件, not the entire directory. You can write more than one path.

#path = Wwwbak

#指定同步忽略的子目录或者文件, support regular.

#ignore = Name *.tmp

#表示保持同步的文件读写权限.

perms =-1

#安静模式

Silent = True

#同步修改时间

Times = True

#force参数表示会以本地所指定文件夹为标准, synchronize the directory to the remote. It is important to note that if the force parameter is specified, then the unison becomes a single synchronization, which means that the folder specified by force is synchronized, similar to rsync.

#force =

#true表示同步时通过文件的创建时间来比较两地文件; False indicates synchronization by comparing the contents of two files.

Fastcheck = False

#links = True

#打开日志记录, specify the log path.

Log = True

LogFile =/home/test/inotify_unison/unison.log

#间隔1秒后, start a new synchronization check

repeat = 1

#失败重试次数

Retry = 10

#指定同步时最大线程数.

MaxThreads = 300

#默认值是true, used to activate rsync transfer mode, FALSE will be transferred using SSH

Rsync = False

#使用ssh的压缩传输方式.

Sshargs =-C

#stream = True

#优化传输参数, the default value is true.

Xferbycopying = True

e) Configure the Unison service on the B-machine

[Email protected]:~$ vim/home/test/.unison/default.prf# unison preferences filebatch = Truegroup = Trueowner = True#path = Wwwbak#ignore = Name *.tmpperms = -1silent = Truetimes = True#force =fastcheck = falselinks = Truelog = Truelogfile =/h Ome/test/inotify_unison/unison.logrepeat = 1retry = 10maxthreads = 300rsync = Falsesshargs =-cstream = truexferbycopying = True

f) Configuring a on-machine feature script

[[email protected] scripts]$ vim /home/test/scripts/inotify_unison.sh#!/bin/bash#  Inotify_unison tools# create by [email protected]# 2014/11/29unison= "/usr/bin/ Unison "local_dir="/home/test/reward_to_sm/"remote_dir="/home/test/reward_to_sm/"remote_user=" Test "ATTRIBUTES=" Modify,delete,create,attrib "# mkdir unison log dirlogdir="/home/test/inotify_unison "#  sync file to many servers  ip_list= "10.8.12.201,10.8.9.9,10.8.12.20" IP_ List= "10.8.12.201" ip_list_file= "/tmp/${user}_inotify_unison_iplist" echo  $IP _list >  $IP _list_ filesed -i  ' s/,/\n/g '   $IP _list_file[ -d /home/test/inotify_unison ] | |  mkdir -p /home/test/inotify_unison# create /tmp/test_inotify_unison.binecho  "#! /bin/bashinotifywait  -mrq --timefmt  '%d/%m/%y %h:%m '  --format  '%T %w% F '  -e ${attributes} ${local_dir} | while read filesdo         For ipaddr in \ ' sed  '/^$/p '   $IP _list_file\ '          do                    $UNISON  default  "${local_dir}"   "Ssh://${remote_user}@\ $IPADDR/${remote_dir}"          donedone " > /tmp/${user}_inotify_unison.binchmod +x  /tmp/${user}_inotify_unison.binstart ()  {     if [ -f /tmp/$ {user}_inotify_unison_lock ]     then           echo -e  "  $USER  inotify_unison had started !\n"   &&  exit 1     fi     echo  "$ User inotify_unison&nBSP;START&NBSP: "     nohup /tmp/${user}_inotify_unison.bin > / tmp/${user}_inotify_unison_nohup.out 2>&1  &     echo  -e  "${user}\n${local_dir}\n${remote_dir}\n${ip_list}"  > /tmp/${user}_inotify_unison_lock} Stop ()  {     echo  $USER  inotify_unison stop  ...  "     rm -f /tmp/${USER}_inotify_unison_lock      ps -ef | grep unison | grep -v grep | grep -v $$  | grep  $USER  | awk  ' {print $2} '  | xargs kill -9} Restart ()  {  stop  start}reload ()  {  stop  start}case  " $ " in    start)         $1         ;;     stop)         $1         ;;     restart)         $1         ;;     reload)         $1         ;;     status)      if [ -f /tmp/${user}_inotify_ unison_lock ]     then           echo -e  "  $USER  inotify_unison had started !\n"            exit 0     else           echo -e    $USER  inotify_unison not  started !\n "          exit 0     fi        ;;     *)         echo $ "Usage: $0 { Start|stop|status|restart|reload} "        exit 2esacexit $?

The Unison configuration file under Ps:windows is located by default in the C:\Documents and Settings\currentuser\.unison directory, and the default profile name is DEFAULT.PRF.

g) Configure the function script on machine b

Consistent with feature scripts on a machine

Modify the Ip_list parameter to a-machine IP

Six, start and close Inotify_unison service

Start command:

su-test/home/test/scripts/inotify_unison.sh start close command: su-test/home/test/scripts/inotify_unison.sh stop

This article is from the "SA for CTO" blog, so be sure to keep this source http://yangwei.blog.51cto.com/5847548/1584585

Use unison + INotify + rsync to implement file-triggered real-time one-way/bidirectional/multidirectional file synchronization