Real-time Data Synchronization for Linux sersync and real-time linuxsersync

Real-time Data Synchronization for Linux sersync and real-time linuxsersync

Sersync uses two software technologies, inotify and rsync, to implement real-time data synchronization. inotify is used to listen to file changes on the server where sersync is located, and uses rsync software to synchronize data, synchronize data to the client server in real time.

Working Process: Enable sersync on the synchronization master server to listen to changes in the file system, then call the rsync command to synchronize the updated files to the target server, and install the sersync software on the master server, install rsync on the target server

1. Client Configuration

[root@localhost2 ~]# cat /etc/rsyncd.conf ##created by cai at 2018-2-24uid=rsyncgid=rsyncuse chroot = nomax connections = 200timeout = 300pid file = /var/run/rsyncd.pidlock file = /var/run/rsync.locklog file = /var/log/rsyncd.log[data]path = /data/ignore errorsread only = falselist = falsehosts allow = 192.168.181.128:52000/24hosts deny = 0.0.0.0/32auth users = rsync_bodysecrets file = /etc/rsync.password

[Root @ localhost2 ~] # Ls-ld/data/
Drwxrwxrwx. 3 rsync 4096 Feb 24/data/

[Root @ localhost2 ~] # Cat/etc/rsync. password
Rsync_body: admin

[Root @ localhost2 ~] # Ls-ld/etc/rsync. password
-Rw -------. 1 root 17 Feb 24/etc/rsync. password #600 permission

[Root @ localhost2 ~] # Netstat-lntup | grep "rsync"
Tcp 0 0 0.0.0.0: 873 0.0.0.0: * LISTEN 1497/rsync
Tcp 0 0: 873: * LISTEN 1497/rsync

2. master server configuration

[root@localhost1 ~]# cat /etc/rsync.password admin[root@localhost1 ~]# ls -ld /etc/rsync.password -rw------- 1 root root 6 Feb 24 03:54 /etc/rsync.password

3. Install the sersync Service

Inotify is used to monitor files. When a file changes, rsync is called to implement real-time synchronization triggered!

Install sersync (note that sersync works on the rsync source server, that is, the client)

[root@salt-client01 ~]# cd /usr/local/src/[root@salt-client01 src]# tar -zxvf sersync2.5.4_64bit_binary_stable_final.tar.gz  -C /usr/local/GNU-Linux-x86/GNU-Linux-x86/sersync2GNU-Linux-x86/confxml.xml[root@salt-client01 src]# cd /usr/local/[root@salt-client01 local]# mv GNU-Linux-x86 sersync[root@salt-client01 local]# cd sersync/[root@salt-client01 sersync]# mkdir conf bin log[root@salt-client01sersync]# mv confxml.xml conf[root@salt-client01sersync]# mv sersync2 bin/sersync

Modify configuration file

* ************************************ 30 rows start ** ************************** <commonParams params = "-artuz"/> #- artuz is the virtual user name (rsync_backup) of the parameter <authstart = "true" users = "rsync during rsync synchronization) "passwordfile =" rsync Password File "/> <userDefinedPort start =" true "port =" 873 "/> <! -- Port = 874 --> <timeout start = "false" time = "100"/> <! -- Timeout = 100 --> <sshstart = "false"/> ************************** **************************** * ****** <failLogpath = "custom log folder (/usr/local/sersync/log) rsync_fail_log.sh "timeToExecute =" 60 "/> <! -- Defaultevery 60 mins execute once --> ******************************** **************************************** * ******* note: if there are multiple directory backups, you can wear multiple configuration files to add them to the-o parameter at startup [root @ salt-client01 conf] # diff confxml. xml confxml. xml. bak24, 25c24, 25 <localpath watch = "/data/"> # data is the local folder to be synchronized to the server directory <remote ip = "192.168.91.166" name = "data"/> # data (server Module name) is the folder on the rsync server, that is, the target folder pushed to the server. Multiple folders can be configured. ---> <localpath watch =" /Opt/tongbu "> <remote ip =" 127.0.0.1 "name =" tongbu1 "/> 31c31 <auth start =" true "users =" rsync_body "passwordfile ="/etc /rsync. password "/> # true to take effect. The virtual account is used during rsync_body synchronization, followed by the password file ---> <auth start = "false" users = "root" passwordfile = "/etc/rsync. pas "/> 33c33 <timeout start =" true "time =" 100 "/> <! -- Timeout = 100 --> # true to take effect ---> <timeout start = "false" time = "100"/> <! -- Timeout = 100 --> 36c36 <failLog path = "/usr/local/sersync/log/rsync_fail_log.sh" timeToExecute = "60"/> <! -- Default every 60 mins execute once --> # Check whether the rsync process is determined and the system does not automatically start ---> <failLog path = "/tmp/rsync_fail_log.sh" timeToExecute = "60"/> <! -- Default evePry 60 mins execute once -->

Start sersync

[Root @ salt-client01 src] # echo 'export PATH = $ PATH: /usr/local/sersync/bin'>/etc/profile # declare the environment variable [root @ salt-client01 src] # source/etc/profile [root @ salt-client01 src] # sersync2- r-d-o/usr/local/sersync/conf/confxml. xml # Start set the system paramexecute: echo 50000000>/proc/sys/fs/inotify/max_user_watchesexecute: echo 327679>/proc/sys/fs/inotify/max_queued_eventsparse the command paramoption: -r rsy Nc all the local files to the remote servers before the sersync workoption:-d run as a daemonoption:-o config xml name:/usr/local/sersync/conf/confxml. xmldaemon thread num: 10 parse xml config filehost ip: localhost host port: 8008 daemon start, sersync run behind the consoleuse rsync password-file: user is rsync_bodypasswordfile is/etc/rsync. passwordconfig xml parse successplease set/etc/rsync D. conf max connections = 0 Manuallysersync working thread 12 = 1 (primary thread) + 1 (fail retry thread) + 10 (daemon sub threads) Max threads numbers is: 22 = 12 (Thread pool nums) + 10 (Sub threads) please according your cpu, use-n param to adjust the cpu rate ------------------------------------------ rsync the directory recursivly to the remote servers onceworking please wait...exe cute command: cd/dat A & rsync-artuz-R -- delete. /-- timeout = 100 rsync_body@192.168.91.166: data -- password-file =/etc/rsync. password>/dev/null 2> & 1run the sersync: watch path is: /data # Now we can see that sersync has been started successfully # detection script [root @ salt-client01 log] # pwd/usr/local/sersync/log [root @ salt-client01 log] # vim rsync_fail_log.sh [root @ salt-client01 log] # chmod + x rsync_fail_log.sh [root @ salt-client01 ~] # Cat/usr/local/sersync/log/rsync_fail_log.sh #! /Bin/bash # Purpose: Check sersync whether it is alive # Author: cai meng zhiSERSYNC = "/usr/local/sersync/bin/sersync2" CONF_FILE = "/usr/local/sersync/conf/confxml. xml "STATUS = $ (ps aux | grep 'sersync2' | grep-v 'grep' | wc-l) if [$ STATUS-eq 0]; then $ SERSYNC-d-r-o $ CONF_FILE & else exit 0; after the fi script is written, add to the scheduled task to */1 *****/bin/bash/usr/local/sersync/log/rsync_fail_log.sh>/dev/null 2> & 1 test synchronization: add a file to the Client [root @ sal T-client01 data] # cp/etc/passwd 192.168.91.156.passwd [root @ salt-client01 data] # lltotal 4-rw-r -- r -- 1 root 1928 Nov 10 192.168.91.156.passwd-rw-r -- 1 root 0 Nov 10 3 Server Detection [root @ salt-master data] # cd/data/[root @ salt-master data] # lltotal 8-rw-r -- r -- 1 root 1928 Nov 10 192.168.91.156.passwd # indicates that it has been synchronized-rw-r -- 1 root 0 Nov 10 3drwxr-xr-x 2 root 4 096 Nov 10 data Client test Delete [root @ salt-client01 data] # rm rf 192.168.91.156.passwd rm: cannot remove 'rf ': No such file or directoryrm: remove regular file '192. 168.91.156.passwd '? Y [root @ salt-client01 data] # lltotal 0-rw-r -- r -- 1 root 0 Nov 10 3 [root @ salt-client01 data] # server side: [root @ salt-master data] # lltotal 4-rw-r -- r -- 1 root 0 Nov 10 3 # Delete drwxr-xr-x 2 root 4096 Nov 10 data

4. Summary of Common Errors

ERROR 1: @ error: auth failed on module xxxxxrsync: connection unexpectedly closed (90 bytes read so far) rsync error: ERROR in rsync protocoldata stream (code 12) at io. c (150) Description: this is because the password is set incorrectly and cannot be successfully logged in. Check rsync. pwd to check whether the customer service matches. This also happens when the rsync service is not started on the server. Error 2: password file must not beother-accessiblecontinuing without password filePassword: Note: this is because rsyncd. pwdrsyncd. sec has incorrect permissions and should be set to 600. For example, chmod600 rsyncd. pwd ERROR 3: @ error: chroot failedrsync: connection unexpectedly closed (75 bytes read so far) rsync error: ERROR in rsync protocoldata stream (code 12) at io. c (150) Description: this is because you are in rsync. the path set in conf does not exist. You need to create a directory to enable synchronization error 4: rsync: failed to connect to218.107.243.2: No route to host (113) rsync error: error in socket IO (code10) at clientserver. c (104) [firewall ER = 2.6.9] Note: as a result of a firewall problem, it is best to completely shut down the firewall first. This is the basic law for troubleshooting, whether it is S or C, the ignore errors option also causes ERROR 5: @ error: access denied to www fromunknown (192.168.1.123) rsync: connection unexpectedly closed (0 bytes received so far) [Cycler] rsync ERROR: error in rsync protocoldata stream (code 12) at io. c (359) Note: this problem is obvious. It is a problem with the hostallow configuration option. beginners like to configure a allowed segment, and the module is the same, resulting in error 6: rsync error: canceled ed SIGINT, SIGTERM, or SIGHUP (code 20) at rsync. c (244) [generator = 2.6.9] rsync error: received SIGUSR1 (code19) at main. c (1182) [handler ER = 2.6.9] Note: Most of the reasons for this problem are that the server service is not properly started. Go to the server and check whether the service is started, check/var/run/rsync. if the pid file exists, the simplest way is to kill the service that has already been started, start the service again, or add the script to the system startup service level, and then shutdown-r now server error 7: rsync: read error: Connection resetby peer (104) rsync error: error in rsync protocoldata stream (code 12) at io. c (604) [sender = 2.6.9] Note: No data exists in the original data directory.