Rsync + inotify for data synchronization and FAQs

Rsync + inotify for data synchronization and FAQs

Rsync: Remote Sync is a data image backup tool for Unix-like systems. Rsync can be used to back up data with low real-time requirements. For example, you can use the specified backup file server to back up data to the specified remote server and regularly perform data mirroring on the local disk.

Inotify: inotify is a file change notification mechanism. inotify allows you to add, delete, modify, and move operations in the monitored file system.

Prepare the environment:

Master server (inotify-Master)	IP: 172.18.42.201
Slave server (inotify-Slave)	IP: 172.18.42.200

1. Deploy inotify-Slave
1. Install the rsync Program
[Root @ inotify-slave ~] # Yum install rsync

2. Create an rsync user
[Root @ inotify-slave ~] # Useradd-s/bin/nologin-M rsync

3. Create a directory for the rsync Module
[Root @ inotify-slave ~] # Mkdir/mydata/# create a Module Directory in rsync Mode
[Root @ inotify-slave ~] # Chown rsync. rsync/mydata/# change the owner and main group so that rsync users have the permission to change data
[Root @ inotify-slave ~] # Ls-ld/mydata/
Drwxr-xr-x 2 rsync 6 May 19 21:02/mydata/

4. Configure the virtual user's password file/etc/rsync. password
[Root @ inotify-slave ~] # Vim/etc/rsync. password
Bkjia: linux # "bkjia" indicates the virtual user name, and "linux" indicates the virtual user password.
[Root @ inotify-slave ~] # Chmod 600/etc/rsync. password # add security to the password file
[Root @ inotify-slave ~] # Ll/etc/rsync. password
-Rw ------- 1 root 12 May 18 21:54/etc/rsync. password

5. Configure the rsync configuration file/etc/rsyncd. conf.
[Root @ inotify-slave ~] # Vim/etc/rsyncd. conf
Uid = rsync
Gid = rsync
User chroot = no
Max connections = 200
Timeout = 300
Read only = no
[Rsync] # define the module name. The name can be random.
Path =/mydata # specify the Module Directory of the rsync user
Auth users = bkjia # specify the virtual user name
Secrets file =/etc/rsync. password # specify the path of the virtual user's password file
Ignore errors # indicates ignore Error

6. Start the rsync service.
[Root @ inotify-slave ~] # Rsync -- daemon -- config =/etc/rsyncd. conf # rsync listens on tcp port 873
[Root @ inotify-slave ~] # Ps aux | grep rsync
Root 1330 0.0 0.0 114640 328? Ss rsync -- daemon -- config =/etc/rsyncd. conf
Root 1338 0.0 0.1 112644 952 pts/0 R + grep -- color = auto rsync

2. Configure the inotify-master password file
1. Configure the password file of the virtual user
[Root @ inotify-master ~] # Echo "linux">/etc/rsync. password
[Root @ inotify-master ~] # Cat/etc/rsync. password
Linux # Note: you only need to write the password of a virtual user. You do not need to write the user name.
[Root @ inotify-master ~] # Chmod 600/etc/rsync. password
[Root @ inotify-master ~] # Ls-ld/etc/rsync. password
-Rw ------- 1 root 6 May 19 21:18/etc/rsync. password

3. Test and push files through inotify-master
[Root @ inotify-master ~] # Echo "Hello Word"> bkjia.txt
[Root @ inotify-master ~] # Rsync-avz bkjia.txt lweim@172.18.42.200: rsync -- password-file =/etc/rsync. password
Sending incremental file list
Bkjia.txt

Sent 81 bytes encoded ed 27 bytes 216.00 bytes/sec
Total size is 11 speedup is 0.10

4. Check inotify-slave's working Module Directory
[Root @ inotify-slave ~] # Ll/mydata/
Total 4
-Rw-r -- 1 rsync 11 May 19 21:21 bkjia.txt
[Root @ inotify-slave ~] # Cat/mydata/bkjia.txt
Hello Word # pushed successfully

5. Deploy rsync-master
1. Use the inofity source code package for compilation and Installation
[Root @ inotify-master ~] # Tar-xf inotify-tools-3.13.tar
[Root @ inotify-master ~] # Cd inotify-tools-3.13/
[Root @ inotify-master inotify-tools-3.13] #./configure -- prefix =/usr/local/inotify # specify the installation path
[Root @ inotify-master ~] # Make-j 4 & make install

2. Write monitoring scripts
12345678910111213141516171819202122 [root @ inotify-master ~] # Vim inotify. sh
#! /Bin/bash

Host = 172.18.42.200 # specify the IP address of inotify-slave
Src =/www/bkjia # directory monitored by the local machine, which can be customized at will
Dst = rsync # inotify-slave rsync User Module Directory
User = bkjia # virtual user name
Passfile =/etc/rsync. password # Call the Local password File
Inotify_home =/usr/local/inotify # specifies the inotify installation directory

If [! -E "$ src"] | [! -E "$ {inotify_home}/bin/inotifywait"] | [! -E "/usr/bin/rsync"] | [! -E "/etc/rsync. password"]; then
# If [! -E "$ {inotify_home}/bin/inotifywait"] | [! -E "/usr/bin/rsync"] | [! -E "/etc/rsync. password"]; then
Echo "Check File and Folder"
Exit 1
Fi


$ {Inotify_home}/bin/inotifywait-mrq-e close_write, delete, create, attrib $ src | while read file
Do
Cd $ src & rsync-arvz-P./-- timeout = 100 $ user @ $ host: $ dst -- password-file = $ passfile &>/dev/null
Done
Exit 0

6. Execute the inotify. sh script on inotify-master.
[Root @ inotify-master ~] # Bash-x inotify. sh # Run the script
+ Host = 172.18.42.200
+ Src =/web/bkjia/
+ Dst = rsync
+ User = bkjia
+ Passfile =/etc/rsync. password
+ Inotify_home =/usr/local/inotify
+ '[''! '-E/web/bkjia/']'
+ '[''! '-E/usr/local/inotify/bin/inotifywait']'
+ '[''! '-E/usr/bin/rsync']'
+ '[''! '-E/etc/rsync. password']'
+ Read file
+/Usr/local/inotify/bin/inotifywait-mrq-e close_write, delete, create, attrib/web/bkjia/

7. Create a file under the inotify-master local monitoring directory
[Root @ inotify-master bkjia] # pwd
/Www/bkjia
[Root @ inotify-master bkjia] # touch a aa aaa aaaa
[Root @ inotify-master bkjia] # ll
Total 0
-Rw-r -- 1 root 0 May 19 22:54
-Rw-r -- 1 root 0 May 19 aa
-Rw-r -- 1 root 0 May 19 22: 54 aaa
-Rw-r -- 1 root 0 May 19 aaaa

8. In the inotify-slave rsync Module Directory, View
[Root @ inotify-slave mydata] # ll
Total 0
-Rw-r -- 1 rsync 0 May 19 22:54
-Rw-r -- 1 rsync 0 May 19 22:54 aa
-Rw-r -- 1 rsync 0 May 19 22:54 aaa
-Rw-r -- 1 rsync 0 May 19 aaaa
[Root @ inotify-slave mydata] # pwd
/Mydata

The following are common problems during rsync Configuration:

Question 1:
@ ERROR: chroot failed
Rsync error: error starting client-server protocol (code 5) at main. c (1522) [runner ER = 3.0.3]

Cause:
The directory on the server does not exist or has no permissions. Creating a directory and correcting permissions can solve the problem.

Question 2:
@ ERROR: auth failed on module tee
Rsync error: error starting client-server protocol (code 5) at main. c (1522) [runner ER = 3.0.3]

Cause:
On the server side, this module (tee) needs to verify the user name and password, but the client does not provide the correct user name and password. Authentication fails.
Provide the correct username and password to solve this problem.

Question 3:
@ ERROR: Unknown module 'TEE _ nonexists'
Rsync error: error starting client-server protocol (code 5) at main. c (1522) [runner ER = 3.0.3]

Cause:
The specified module does not exist on the server. Provide the correct module name or modify it on the server to the module you want to solve the problem.

Question 1:
Problems encountered on the client:
Rsync-auzv -- progress -- password-file =/etc/rsync. pas root@192.168.133.128: backup/home/
Rsync: cocould not open password file "/etc/rsync. pas": No such file or directory (2)
Password:
@ ERROR: auth failed on module backup
Rsync error: error starting client-server protocol (code 5) at main. c (1506) [runner ER = 3.0.7]
This problem occurs: the file/etc/rsync. pas is not set on the client, but this parameter is added when the rsync command is used --
Password-file =/etc/rsync. pas

Question 2:
Rsync-auzv -- progress -- password-file =/etc/rsync. pas root@192.168.133.128: backup/home/
@ ERROR: auth failed on module backup
Rsync error: error starting client-server protocol (code 5) at main. c (1506) [runner ER = 3.0.7]
This problem occurs: the file/etc/rsync. pas has been set on the client, and the password 111111 is set in it, which is consistent with that on the server,
The server segment is set incorrectly. The server side should set/etc/rsync. pas, which contains root: 111111. the login name is indispensable here.

Question 3:
Rsync-auzv -- progress -- password-file =/etc/rsync. pas root@192.168.133.128: backup/home/
@ ERROR: chdir failed
Rsync error: error starting client-server protocol (code 5) at main. c (1506) [runner ER = 3.0.7]
This problem occurs because the/home/backup directory on the server is not set, so the prompt is: chdir failed

Question 4:
Rsync: write failed on "/home/backup2010/wensong": No space left on device (28)
Rsync error: error in file IO (code 11) at handler er. c (302) [handler ER = 3.0.7]
Rsync: connection unexpectedly closed (2721 bytes encoded Ed so far) [generator]
Rsync error: error in rsync protocol data stream (code 12) at io. c (601) [generator = 3.0.7]
Unable to operate because the disk space is insufficient.
You can use df/home/backup2010 to view available and used space.

Question 5: network collection problems
1. Permission issues
A message similar to the following is displayed: rsync: opendir "/kexue" (in dtsChannel) failed: Permission denied (13) check whether the synchronization directory Permission is 755.
2. time out
Rsync: failed to connect to 203.100.192.66: Connection timed out (110)
Rsync error: error in socket IO (code 10) at clientserver. c (124) [handler ER = 3.0.5]
Check the server port netstat-tunlp for remote telnet testing.
Communication may fail because the firewall on the client or server is enabled. You can set rules to allow rsync (Port 873) or directly disable the firewall.

Another way is to prompt that you do not have the permission during the synchronization process. (add SvcwRsync to the synchronization directory and set SvcwRsync as the administrator)

3. The service is not started.
Rsync: failed to connect to 10.10.10.170: Connection refused (111)
Rsync error: error in socket IO (code 10) at clientserver. c (124) [handler ER = 3.0.5]
Start the service: rsync -- daemon -- config =/etc/rsyncd. conf
4. Full disk space
Rsync: recv_generator: mkdir "/teacherclubBackup/rsync ...... "Failed: No space left on device (28)
* ** Skipping any contents from this failed directory ***
5. Ctrl + C or a large number of files
Rsync error: canceled ed SIGINT, SIGTERM, or SIGHUP (code 20) at rsync. c (544) [Cycler = 3.0.5]
Rsync error: canceled ed SIGINT, SIGTERM, or SIGHUP (code 20) at rsync. c (544) [generator = 3.0.5]
Note: Most of the reasons for this problem are that the server service is not started properly. Go to the server and check whether the service is started. Then, check/var/run/rsync. if the pid file exists, the simplest way is to kill the started service, then start the service again, or add the script to the system startup service level, and then shutdown the-r now server.

6. xnetid startup
Rsync: read error: Connection reset by peers (104)
Rsync error: error in rsync protocol data stream (code 12) at io. c (759) [runner ER = 3.0.5]
View rsync logs
Rsync: unable to open configuration file "/etc/rsyncd. conf": No such file or directory
The default location of the configuration file for xnetid search is/etc, and a soft link is created based on the actual situation. For example:
Ln-s/etc/rsyncd. conf/etc/rsyncd. conf
You can also change the path of the specified default configuration file in the/etc/xinetd. d/rsync configuration file.
Rsync configure:
Configuration 1:
Ignore errors
Note: It is best to add this option. Otherwise, errors often occur in many crontab tasks, which you cannot see every day, the probability of this error is relatively high, because the disk IO is a bottleneck for any large project or system.

Rsync error:
Error 1:
@ ERROR: auth failed on module xxxxx
Rsync: connection unexpectedly closed (90 bytes read so far)
Rsync error: error in rsync protocol data stream (code 12) at I/O. c (150)
Note: this is because the password is set incorrectly and cannot be successfully logged in. Check rsync. pwd to see if the customer service matches. This also happens when the rsync service is not started on the server.
Error 2:
Password file must not be other-accessible
Continuing without password file
Password:
Note: this is because rsyncd. pwd rsyncd. sec has incorrect permissions and should be set to 600. Example: chmod 600 rsyncd. pwd
Error 3:
@ ERROR: chroot failed
Rsync: connection unexpectedly closed (75 bytes read so far)
Rsync error: error in rsync protocol data stream (code 12) at I/O. c (150)
Note: this is because the path you set in rsync. conf does not exist. You need to create a new directory to enable synchronization.
Error 4:
Rsync: failed to connect to 218.107.243.2: No route to host (113)
Rsync error: error in socket IO (code 10) at clientserver. c (104) [handler ER = 2.6.9]
Note: As a result of a firewall problem, it is best to completely shut down the firewall first. This is the basic principle for troubleshooting. problems such as S, C, and ignore errors can also result in problems.

Error 5:
@ ERROR: access denied to www from unknown (192.168.1.123)
Rsync: connection unexpectedly closed (0 bytes encoded Ed so far) [Cycler]
Rsync error: error in rsync protocol data stream (code 12) at I/O. c (359)
Note: This problem is obvious. It is the problem of configuring the host allow option. beginners like to configure a allowed segment, and the module is the same, resulting in
Error 6:
Rsync error: lost Ed SIGINT, SIGTERM, or SIGHUP (code 20) at rsync. c (244) [generator = 2.6.9]
Rsync error: lost Ed SIGUSR1 (code 19) at main. c (1182) [author ER = 2.6.9]
Note: Most of the reasons for this problem are that the server service is not started properly. Go to the server and check whether the service is started. Then, check/var/run/rsync. if the pid file exists, the simplest way is to kill the started service, then start the service again, or add the script to the system startup service level, and then shutdown the-r now server.
Error 7:
Rsync: read error: Connection reset by peers (104)
Rsync error: error in rsync protocol data stream (code 12) at io. c (604) [sender = 2.6.9]
Note: No data exists in the original data directory.

RSync for file backup Synchronization

Monitor host files and directories using inotifywait

Using inotify + rsync for Linux File batch update

Inotify-tools + rsync real-time file synchronization installation and configuration

Complete rsync synchronization Configuration

Remote synchronization of Rsync in CentOS 6.5

Differential file comparison and extraction in Linux folders-the use of rsync

Rsync details: click here
Rsync: click here

This article permanently updates the link address: