In centos6, Rsync + sersync achieves real-time data synchronization (Big Data high-performance solution)

Source: Internet
Author: User
Tags inotify rsync

Install sersync to trigger rsync in real time for synchronization.

1. Check whether the server kernel supports inotify.
Ll/proc/sys/fs/inotify # list the file directories. The following content appears, indicating that the server kernel supports inotify.
-Rw-r -- 1 root 0 Mar 7 0:17 max_queued_events
-Rw-r -- 1 root 0 Mar 7 max_user_instances
-Rw-r -- 1 root 0 Mar 7 max_user_watches
Note: In Linux, the minimum kernel that supports inotify is 2.6.13. You can run the following command: uname-a to view the kernel:
CentOS 5.X kernel 2.6.18. inotify is supported by default.

2. Modify the default inotify parameter (the default inotify kernel parameter value is too small)
View the default system parameter values:
Sysctl-a | grep max_queued_events
The result is: fs. inotify. max_queued_events = 16384.
Sysctl-a | grep max_user_watches
The result is: fs. inotify. max_user_watches = 8192.
Sysctl-a | grep max_user_instances
Result: fs. inotify. max_user_instances = 128
Modify parameters:
Sysctl-w fs. inotify. max_queued_events = "99999999"
Sysctl-w fs. inotify. max_user_watches = "99999999"
Sysctl-w fs. inotify. max_user_instances = "65535"
Vi/etc/sysctl. conf # add the following code
Fs. inotify. max_queued_events = 99999999
Fs. inotify. max_user_watches = 99999999
Fs. inotify. max_user_instances = 65535
: Wq! # Save and exit

Parameter description:
Max_queued_events:
The maximum length of the inotify Queue. If the value is too small, the error "** Event Queue Overflow **" will occur, resulting in inaccurate monitoring files.
Max_user_watches:
The number of directories in the file to be synchronized can be found/home/www.osyunwei.com-type d | wc-l, make sure that the max_user_watches value is greater than the statistical result (here/home/www.osyunwei.com is the directory for synchronizing files)
Max_user_instances:
Maximum value of inotify instance created by each user

3. Install sersync
Sersync download (Download address Baidu)
Upload sersync2.5.4_64bit_binary_stable_final.tar.gz to the/usr/local/src directory.
Cd/usr/local/src
Tar zxvf sersync2.5.4_64bit_binary_stable_final.tar.gz # decompress
Mv GNU-Linux-x86/usr/local/sersync # Move the directory to/usr/local/sersync

4. Configure sersync
Cd/usr/local/sersync # enter the sersync installation directory
Cp confxml. xml confxml. xml-bak # back up the original file
Vi confxml. xml # Edit and modify the following code
----------------------------------- Code start -----------------------------------------------
<? Xml version = "1.0" encoding = "ISO-8859-1"?>
<Head version = "2.5">
<Host hostip = "localhost" port = "8008"> <Debug start = "false"/>
<FileSystem xfs = "false"/>
<Filter start = "false">
<Exclude expression = "(. *) \. svn"> </exclude>
<Exclude expression = "(. *) \. gz"> </exclude>
<Exclude expression = "^ info/*"> </exclude>
<Exclude expression = "^ static/*"> </exclude>
</Filter>
<Inotify>
<Delete start = "true"/>
<CreateFolder start = "true"/>
<CreateFile start = "false"/>
<CloseWrite start = "true"/>
<MoveFrom start = "true"/>
<MoveTo start = "true"/>
<Attrib start = "false"/>
<Modify start = "false"/>
</Inotify>

<Sersync>
<Localpath watch = "/home/DemoRoot/751/Uploads/">
<Remote ip = "219.234.7.31" name = "home_www"/>
<! -- <Remote ip = "192.168.8.39" name = "tongbu"/> -->
</Localpath>
<Rsync>
<CommonParams params = "-artuz"/>
<Auth start = "true" users = "root" passwordfile = "/etc/passwd.txt"/>
<UserDefinedPort start = "false" port = "874" type = "codeph" text = "codeph"/> <! -- Port = 874 -->
<Timeout start = "false" time = "100"/> <! -- Timeout = 100 -->
<Ssh start = "false"/>
</Rsync>
<FailLog path = "/tmp/rsync_fail_log.sh" timeToExecute = "60"/> <! -- Default every 60 mins execute once -->
<Crontab start = "true" schedule = "600"> <! -- 600mins -->
<Crontabfilter start = "false">
<Exclude expression = "*. php"> </exclude>
<Exclude expression = "info/*"> </exclude>
</Crontabfilter>
</Crontab>
<Plugin start = "false" name = "command"/>
</Sersync>

<Plugin name = "command">
<Param prefix = "/bin/sh" suffix = "" ignoreError = "true"/> <! -- Prefix/opt/tongbu/mmm. sh suffix -->
<Filter start = "false">
<Include expression = "(. *) \. php"/>
<Include expression = "(. *) \. sh"/>
</Filter>
</Plugin>

<Plugin name = "socket">
<Localpath watch = "/home/DemoRoot/751/Uploads/">
<Deshost ip = "192.168.138.20" port = "8009"/>
</Localpath>
</Plugin>
<Plugin name = "refreshCDN">
<Localpath watch = "/data0/htdocs/cms.xoyo.com/site/">
<Cdninfo domainname = "ccms.chinache.com" port = "80" username = "xxxx" passwd = "xxxx"/>
<Sendurl base = "http://pic.xoyo.com/cms"/>
<Regexurl regex = "false" match = "cms.xoyo.com/site ([/a-zA-Z0-9] *) .xoyo.com/images"/>
</Localpath>
</Plugin>
</Head>
----------------------------------- Code end -----------------------------------------------
: Wq! # Save and exit

Parameter description:
Localpath watch = "/home/DemoRoot/751/Uploads/": # synchronization Directory of the source server
219.234.7.31: # IP address of the target server
Name = "home_www": # name of The rsync synchronization directory module on the target server
Users = "root": # target server rsync user name
Passwordfile = "/etc/passwd.txt": # the target server rsync synchronizes the user's password to the storage path of the source server
Remote ip = "219.234.7.31": # ip address of the target server, one per line
FailLog path = "/tmp/rsync_fail_log.sh" # Log of script running failure
Start = "true" # set to true to execute full synchronization every 600 minutes

5. Set sersync to monitor automatic execution upon startup
Vi/etc/rc. d/rc. local # edit and add the last line
/Usr/local/sersync/sersync2-d-r-o/usr/local/sersync/confxml. xml # set the script to run automatically upon startup
: Wq! # Save and exit

6. Add a script to monitor whether sersync runs properly
Create the/opt/check_sersync.sh file first
Vi/opt/check_sersync.sh # edit and add the following code:
----------------------------------- Code start -----------------------------------------------
#! /Bin/sh
Sersync = "/usr/local/sersync/sersync2"
Confxml = "/usr/local/sersync/confxml. xml"
Status = $ (ps aux | grep 'sersync2' | grep-v 'grep' | wc-l)
If [$ status-eq 0];
Then
$ Sersync-d-r-o $ confxml &
Else
Exit 0;
Fi
----------------------------------- Code end -www.111cn.net ----------------------------------------------
: Wq! # Save and exit

Chmod-R 777/opt/check_sersync.sh # add the script execution permission

Vi/etc/crontab # edit and add the following line at the end

*/5 * root/home/crontab/check_sersync.sh>/dev/null 2> & 1 # execute the script every 5 minutes

Service crond reload # reload the service

6. Test whether the rsync synchronization script is normal when sersync is triggered in real time.
Add files to the source server and check whether the target server is synchronized.

If all the tests are successful, the inotify trigger rsync synchronization script in real time runs normally.

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.