Sersync synchronizes data in real time and sersync data in real time
1.1 first milestone: Install sersync software 1.1.1 upload the software to the server and decompress it
1. Upload the software to the server rz-E
To facilitate the management of the upload location, set it to/server/tools.
2. decompress the Software Package
[Root @ backup sersync_installdir_64bit] # tree
.
── Sersync
── Bin
│ ── Sersync
── Conf
│ ── Confxml. xml
── Logs
1.1.2 Binary Package Installation MethodHow to install the software in a binary package (how to install the green software ):
Directly decompress the package.
[Root @ nfs01 sersync_installdir_64bit] # mv sersync // usr/local/
[Root @ nfs01 tools] # tree/usr/local/sersync/
/Usr/local/sersync/
── Bin
│ ── Sersync
── Conf
│ ── Confxml. xml
── Logs
3 directories, 2 files
1.2 second milestone: Prepare sersync configuration file 1.2.1 common syntax formatsRsync configuration file writing: ini syntax
Sersync configuration file writing: xml syntax
Ansible configuration file Compilation: yml syntax
1.2.2 modify the configuration fileBackup before writing
[Root @ backup conf] # ll
Total 4
-Rw-r -- 1 root 2214 Oct 26 2011 confxml. xml
[Root @ backup conf] # cp confxml. xml {,. bak}
6-11 rows indicate that the synchronized data is excluded, which is equivalent to the -- exclude function.
6 <filter start = "false">
7 <exclude expression = "(. *) \. svn"> </exclude>
8 <exclude expression = "(. *) \. gz"> </exclude>
9 <exclude expression = "^ info/*"> </exclude>
10 <exclude expression = "^ static/*"> </exclude>
11 </filter>
Line 12-21 uses the inotify function to monitor specified events, which is equivalent to-e create, delete ...... Indicates the specified monitoring event information.
12 <inotify>
13 <delete start = "true"/>
14 <createFolder start = "true"/>
15 <createFile start = "false"/>
16 <closeWrite start = "true"/>
17 <moveFrom start = "true"/>
18 <moveTo start = "true"/>
19 <attrib start = "false"/>
20 <modify start = "false"/>
21 </inotify>
Row 24-28: Where is the push? name = module is the address of the rsync server
24 <localpath watch = "/data"> # monitor the Directory
25 <remote ip = "172.16.1.41" name = "backup"/>
26 <! -- <Remote ip = "192.168.8.39" name = "tongbu"/> -->
27 <! -- <Remote ip = "192.168.8.40" name = "tongbu"/> -->
28 </localpath>
Line 29-35 defines the parameter information during rsync push.
Note: Do not make spelling mistakes.(True). Otherwise, the program cannot be started normally.
29 <rsync>
30 <commonParams params = "-az"/>
31 <auth start = "true" users = "rsync_backup" passwordfile = "/etc/rsync. password"/>
32 <userDefinedPort start = "false" port = "874" type = "codeph" text = "codeph"/> <! -- Port = 874 -->
33 <timeout start = "false" time = "100" type = "codeph" text = "codeph"/> <! -- Timeout = 100 -->
34 <ssh start = "false"/>
35 </rsync>
Final content of the configuration file:
[Root @ nfs01 tools] # cat/usr/local/sersync/conf/confxml. xml
<? Xml version = "1.0" encoding = "ISO-8859-1"?>
<Head version = "2.5">
<Host hostip = "localhost" port = "8008">
<Debug start = "false"/>
<FileSystem xfs = "false"/>
<Filter start = "false">
<Exclude expression = "(. *) \. svn"> </exclude>
<Exclude expression = "(. *) \. gz"> </exclude>
<Exclude expression = "^ info/*"> </exclude>
<Exclude expression = "^ static/*"> </exclude>
</Filter>
<Inotify>
<Delete start = "true"/>
<CreateFolder start = "true"/>
<CreateFile start = "false"/>
<CloseWrite start = "true"/>
<MoveFrom start = "true"/>
<MoveTo start = "true"/>
<Attrib start = "false"/>
<Modify start = "false"/>
</Inotify>
<Sersync>
<Localpath watch = "/data">
<Remote ip = "172.16.1.41" name = "nfsbackup"/>
<! -- <Remote ip = "192.168.8.39" name = "tongbu"/> -->
<! -- <Remote ip = "192.168.8.40" name = "tongbu"/> -->
</Localpath>
<Rsync>
<CommonParams params = "-az"/>
<Auth start = "true" users = "rsync_backup" passwordfile = "/etc/rsync. password"/>
<UserDefinedPort start = "false" port = "874" type = "codeph" text = "codeph"/> <! -- Port = 874 -->
<Timeout start = "false" time = "100"/> <! -- Timeout = 100 -->
<Ssh start = "false"/>
</Rsync>
<FailLog path = "/tmp/rsync_fail_log.sh" timeToExecute = "60"/> <! -- Default every 60 mins execute once -->
<Crontab start = "false" schedule = "600"> <! -- 600mins -->
<Crontabfilter start = "false">
<Exclude expression = "*. php"> </exclude>
<Exclude expression = "info/*"> </exclude>
</Crontabfilter>
</Crontab>
<Plugin start = "false" name = "command"/>
</Sersync>
<Plugin name = "command">
<Param prefix = "/bin/sh" suffix = "" ignoreError = "true"/> <! -- Prefix/opt/tongbu/mmm. sh suffix -->
<Filter start = "false">
<Include expression = "(. *) \. php"/>
<Include expression = "(. *) \. sh"/>
</Filter>
</Plugin>
<Plugin name = "socket">
<Localpath watch = "/opt/tongbu">
<Deshost ip = "192.168.138.20" port = "8009"/>
</Localpath>
</Plugin>
<Plugin name = "refreshCDN">
<Localpath watch = "/data0/htdocs/cms.xoyo.com/site/">
<Cdninfo domainname = "ccms.chinache.com" port = "80" username = "xxxx" passwd = "xxxx"/>
<Sendurl base = "http://pic.xoyo.com/cms"/>
<Regexurl regex = "false" match = "cms.xoyo.com/site ([/a-zA-Z0-9] *) .xoyo.com/images"/>
</Localpath>
</Plugin>
</Head>
1.3 milestone 3: Start sersync 1.3.1 Modify file permissions (executable)First, let the program give the File Execution permission
[Root @ nfs01 bin] # chmod a + x sersync
[Root @ nfs01 bin] # ll
Total 1768
-Rwxr-xr-x 1 root 1810128 Oct 26 2011 sersync
1.3.2 view the help information of the software[Root @ nfs01 bin] #./sersync-h
Set the system param
Execute: echo 50000000>/proc/sys/fs/inotify/max_user_watches
Execute: echo 327679>/proc/sys/fs/inotify/max_queued_events
Parse the command param
_______________________________________________________
Important parameter-d: Enable daemon mode
Important parameter-r: Push the monitoring directory and remote host with the rsync command before monitoring
Parameter-n: specifies the number of daemon threads enabled. The default value is 10.
Important parameter-o: Specifies the configuration file. By default, the confxml. xml file is used.
Parameter-m: other modules are enabled separately. Use-m refreshCDN to enable and refresh the CDN module.
Parameter-m: other modules are enabled separately. Use-m socket to enable the socket module.
Parameter-m: enable other modules separately. Enable the http module using-m http.
If the-m parameter is not added, the synchronization program is executed by default.
________________________________________________________________
1.3.3 start the program in the bin directory of the program./Sersync-dro/usr/local/sersync/conf/confxml. xml
1.3.4 method 2Add the bin directory of/usr/local/sersync/bin/program to PATH.
Export PATH =/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin: /usr/local/sersync/bin/
Then the sersync command can be used directly.
[Root @ nfs01 scripts] # sersync-dro/usr/local/sersync/conf/confxml. xml
Set the system param
Execute: echo 50000000>/proc/sys/fs/inotify/max_user_watches
Execute: echo 327679>/proc/sys/fs/inotify/max_queued_events
Parse the command param
Option:-d run as a daemon
Option:-r rsync all the local files to the remote servers before the sersync work
Option:-o config xml name:/usr/local/sersync/conf/confxml. xml
Daemon thread num: 10
Parse xml config file
Host ip: localhost host port: 8008
Daemon start, sersync run behind the console
Use rsync password-file:
User is rsync_backup
Passwordfile is/etc/rsync. password
Config xml parse success
Please set/etc/rsyncd. conf max connections = 0 Manually
Sersync working thread 12 = 1 (primary thread) + 1 (fail retry thread) + 10 (daemon sub threads)
Max threads numbers is: 22 = 12 (Thread pool nums) + 10 (Sub threads)
Please according your cpu, use-n param to adjust the cpu rate
------------------------------------------
Rsync the directory recursivly to the remote servers once
Working please wait...
Execute command: cd/data & rsync-az-R -- delete. /rsync_backup@172.16.1.41: nfsbackup -- password-file =/etc/rsync. password>/dev/null 2> & 1
Run the sersync:
Watch path is:/data
1.4 Inotify and sersync summary and comparison 1.4.1 real-time Inotify concurrency:Conclusion: after testing, 200 concurrent files per second, almost no latency in data synchronization (less than 1 second)
1.4.2 inotify advantages:1) monitors file system event changes and synchronizes real-time data through synchronization tools.
1.4.3 inotify disadvantages1) if there are more than 200 concurrent files (10-100 k), synchronization will be delayed.
2) The script we wrote earlier is pushed once every time, but it is indeed incremental. You can also synchronize only the changed files without changing them.
3) after the event is monitored, calling rsync for synchronization is a single process, while sersync for multi-process synchronization. Now that inotify-tools is available, why should we develop sersync?
1.4.4 many serysync functions: (inotify + rsync command)1) supports configuration file management
2) Real daemon socket
3) scheduled retransmission of failed files (scheduled task function)
4) third-party HTTP interfaces (for example, updating cdn cache)
5) default multi-process rsync Synchronization
1.4.5 Summary of High-concurrency Real-time Data Synchronization solutions:1) inotify (sersync) + rsync is at the file level.
2) drbd file system level, file system level, block-based synchronization. Disadvantages: Backup node data unavailable
3) Synchronization functions of third-party software: mysql synchronization (master-slave replication), oracle, mongodb
4) Dual-Write Program, directly write two servers.
5) solve the problem by using the product business logic (read/write splitting, backup cannot be read, read Master)
Start building with 50+ products and up to 12 months usage for Elastic Compute Service
1 on 1 presale consultation
24/7 Technical Support 6 Free Tickets per Quarter Faster Response
Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.
A comprehensive suite of global cloud computing services to power your business
Payment Methods We Support
