Real-time file synchronization based on rsync + inotify

Real-time file synchronization based on rsync + inotify

Rsync: remote synchronization, which allows you to quickly copy and synchronize data between local and remote hosts. When copying data, rsync compares the source data with the target data, incremental copy

Listening on 873/tcp
Rsync has many options:
-N: when you are not sure whether the command can be executed as required, you must test it in advance.-n can complete this function;
-V: -- verbose, detailed output mode
-Q: -- quiet. The silent mode has as few output as possible.
-C: -- checksum. enable the verification function to force file transfer verification.
-R: -- recursive, recursive replication;
-A: -- archives, archive, retain the original attributes of the file
-P: -- perms reserves the File Permission.
-T: -- times indicates the timestamp of the file retained.
-L: -- links retains the symbolic link of the file.
-G: -- group indicates the group of objects retained.
-O: -- owner retains the owner of the file
-D: -- devices: Keep the Device File

-E ssh: indicates using the ssh protocol for bearer
-Z: transfer after File compression

-- Progress: display progress bar
-- Stats: shows how to perform compression and transmission.

Test before copying:

[Root @ martin ~] # Ls
Anaconda-ks.cfg d1 d2 install. log install. log. syslog
[Root @ martin ~] # Rsync install. log d2-n
[Root @ martin ~] # Rsync install. log. a d2-n
Rsync: link_stat "/root/install. log. a" failed: No such file or directory (2)
Rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main. c (1039) [sender = 3.0.6]

Note: When the rsync command is in use, if there is a diagonal line at the end of the Source parameter, the content of the specified directory will be copied instead of the Directory itself; if there is no diagonal line, the directory itself will be copied; the diagonal line at the end of the target parameter does not work.
No/test after source data
[Root @ martin ~] # Rsync-rv d1 d2
Sending incremental file list
D1/
D1/a. bin. SQL
D1/a. SQL
D1/all. SQL
D1/kk/a. SQL

Sent 658784 bytes encoded ed 109 bytes 1317786.00 bytes/sec
Total size is 658357 speedup is 1.00
[Root @ martin ~] # Ls d2/
D1

The source data is followed by/test:
[Root @ martin ~] # Rsync-rv d1/d2
Sending incremental file list
A
A. bin. SQL
A. SQL
All. SQL
Kk/
Kk/a. SQL

Sent 658777 bytes encoded ed 111 bytes 1317776.00 bytes/sec
Total size is 658357 speedup is 1.00
[Root @ martin ~] # Ls d2
A. bin. SQL a. SQL all. SQL d1 kk

Ssh-based remote push:
[Root @ martin d2] # rsync-e "ssh-p6789"-r ../d1 root @ sherry :~ /-- Progress

[Root @ sherry ~] # Ls
A. SQL anaconda-ks.cfg d1 install. log install. log. syslog

Remote pulling Based on ssh:
[Root @ martin ~] # Rsync-e "ssh-p6789"-rv root @ sherry :~ /D3. -- progress
Root @ sherry's password:
Refreshing incremental file list
D3/
D3/a. SQL
1868 100% 1.78 MB/s 0:00:00 (xfer #1, to-check = 0/2)

Sent 34 bytes encoded ed 1972 bytes 802.40 bytes/sec
Total size is 1868 speedup is 0.93
[Root @ martin ~] # Ls
Anaconda-ks.cfg d1 d2 d3 install. log install. log. syslog

Installation:
Server: sherry
Client: martin
[Root @ sherry ~] # Yum install xinetd rsync

Configuration file:
The configuration file is/etc/rsyncd. conf. for help, Run man rsyncd. conf.
Define one global configuration and multiple rsync sharing configurations

[Root @ sherry ~] # Vim/etc/rsyncd. conf
# Global Settings
Uid = nobody
Gid = nobody
Use chroot = no
Max connections = 10
Strict modes = yes
Pid file =/var/run/rsyncd. pid
Log file =/var/log/rsyncd. log
# Directory to be synced
[Synced_name]
Path =/path/to/some_dir # shared directory
Ignore errors = yes # ignore errors during Synchronization
Read only = no # yes, it cannot be pushed and can only be downloaded
Write only = no # yes, pull cannot upload only
Hosts allow = white_list_ip/net # whitelist
Hosts deny = * # blacklist

Note:
1. Both are unavailable. Access is allowed by default;
2. Only hosts allow is displayed: The whitelist is defined, but the host that is not matched is processed by default rules, that is, allowed;
3. Only hosts deny is displayed: The Blacklist is defined; all of the blacklists are rejected;
4. Both of them appear at the same time: Check hosts allow first, if it matches allow; otherwise, check hosts deny. If it matches, it is rejected. If there is no match between the two, it is processed by the default rule, that is, allow;

List = false # list Directories
Uid = root # Do not use root
Gid = root
Auth users = username # User Authentication
Secrets file =/etc/rsyncd. passwd # file format (plaintext): username: password file Permission must be set to 600;

[Root @ sherry rsync] # cat/etc/rsyncd. conf
# Global Settings
Uid = nobody
Gid = nobody
Use chroot = no
Max connections = 10
Strict modes = yes
Pid file =/var/run/rsyncd. pid
Log file =/var/log/rsyncd. log

[Mysql]
Path =/mydata/backup/rsync-mysql
Ignore errors = yes
Read only = no
Write only = no
Hosts allow = 192.168.1.0/24
Hosts deny = *
List = false
Uid = sherry
Gid = sherry
Auth users = tom
Secrets file =/etc/rsyncd. passwd

[Root @ sherry rsync-mysql] # pwd
/Mydata/backup/rsync-mysql
[Root @ sherry backup] # chown sherry. sherry rsync-mysql/# An error is reported when setfacl is used for data consistency.

Add account password:
[Root @ sherry rsync] # cat/etc/rsyncd. passwd
Tom: 222222.
[Root @ sherry rsync] # chmod 600/etc/rsyncd. passwd

Start Port:
[Root @ sherry rsync] # chkconfig xinetd on
[Root @ sherry rsync] # service xinetd restart
[Root @ sherry rsync] # chkconfig rsync on
[Root @ sherry rsync] # ss-lntup | grep 873
Tcp LISTEN 0 64: 873: * users :( ("xinetd", 121111,5 ))

Test:
Push manually enter password:
# Client
[Root @ martin mysql] # touch
[Root @ martin mysql] # ls
A
[Root @ martin mysql] # rsync a tom @ sherry: mysql
Password:
# Server
[Root @ sherry rsync-mysql] # ll
Total 0
-Rw-r -- 1 sherry 0 May 28 10:01

Push uses files instead of passwords:
# Client
[Root @ martin ~] # Vim/etc/rsyncd. passwd
222222
[Root @ martin ~] # Chmod 600/etc/rsyncd. passwd
[Root @ martin mysql] # touch B
[Root @ martin mysql] # rsync -- password-file =/etc/rsyncd. passwd B tom @ sherry: mysql
# Server
[Root @ sherry rsync-mysql] # ll
Total 0
-Rw-r -- 1 sherry 0 May 28 10:01
-Rw-r -- 1 sherry 0 May 28 10:03 B

Pull:
[Root @ martin mysql] # rm-fr *
[Root @ martin mysql] # rsync -- password-file =/etc/rsyncd. passwd tom @ sherry: mysql /*.
[Root @ martin mysql] # ls
A B

Incremental upload: (upload files not available on the server)
# Client
[Root @ martin mysql] # rm-f *
[Root @ martin mysql] # ls
[Root @ martin mysql] # touch a B
[Root @ martin mysql] # ls
A B
[Root @ martin mysql] # rsync -- password-file =/etc/rsyncd. passwd-az./tom @ sherry: mysql
# Server
[Root @ sherry rsync-mysql] # rm-f *
[Root @ sherry rsync-mysql] # ls
A B

# Client
[Root @ martin mysql] # rm-f B
[Root @ martin mysql] # touch c
[Root @ martin mysql] # ls
A c
[Root @ martin mysql] # rsync -- password-file =/etc/rsyncd. passwd-az./tom @ sherry: mysql
# Server
[Root @ sherry rsync-mysql] # ls
A B c

Synchronous upload: (Data Consistency)
12345678910111213 # client
[Root @ martin mysql] # touch d
[Root @ martin mysql] # ls
A c d
# Server
[Root @ sherry rsync-mysql] # ls
A B c

# Client
[Root @ martin mysql] # rsync -- password-file =/etc/rsyncd. passwd-az -- delete./tom @ sherry: mysql
# Server
[Root @ sherry rsync-mysql] # ls
A c d

Synchronous download: (Data Consistency)
# Server
[Root @ sherry rsync-mysql] # rm-f c
[Root @ sherry rsync-mysql] # touch g
[Root @ sherry rsync-mysql] # ls
A d g
# Client
[Root @ martin mysql] # ls
A c d
[Root @ martin mysql] # rsync -- password-file =/etc/rsyncd. passwd-az -- delete tom @ sherry: mysql ./
[Root @ martin mysql] # ls
A d g

Linux Kernel supports inotify starting from 2.6.13. inotify can be used to monitor file systems for adding, deleting, modifying, and moving events. Inotify is an event mechanism that provides a real-time event response mechanism for the application file system. Compared with the cron round robin mechanism, inotify consumes less resources.

:
Http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz

Installation:
[Root @ martin inotify-tools-3.14] #./configure -- prefix =/usr/local/inotify-tools-3.14
[Root @ martin inotify-tools-3.14] # make & make install
[Root @ martin local] # ln-sv/usr/local/inotify-tools-3.14 // usr/local/inotify-tools

[Root @ martin bin] # cd/usr/local/inotify-tools/bin
[Root @ martin bin] # ls
Inotifywait inotifywatch

# Inotifywait waits for a specific File System Event (open, close, delete, etc.) on the monitored file or directory and is blocked after execution. It is suitable for shell scripts.
# Inotifywatch collects statistics on the usage of monitored file systems, which indicates the number of times a file system event occurs.

Inotifywait
-R | -- recursive query directory
Q | -- quiet Mode of quiet, with minimal output
-M | -- monitor always keeps the event listening status
-- Excludei is case insensitive when excluding files or directories.
-- Timefmt specifies the time output format
-- Format specifies the output format of information during command execution. % T indicates the previous time format.
-E event

Monitoring: (generally, you can delete the three items, create, and modify close_write)
[Root @ martin bin] #. /inotifywait-mrq -- timefmt '% d/% m/% y % H: % m' -- format' % T % w % F'-e delete, create, close_write/backup/mysql/
# Create a file under/backup/mysql/in another window
[Root @ martin mysql] # touch c
[Root @ martin bin] #. /inotifywait-mrq -- timefmt '% d/% m/% y % H: % m' -- format' % T % w % F'-e delete, create, close_write/backup/mysql/
28/05/16/backup/mysql/c
28/05/16/backup/mysql/c # Trigger create close_write twice


# Delete
[Root @ martin mysql] # rm-f
[Root @ martin bin] #. /inotifywait-mrq -- timefmt '% d/% m/% y % H: % m' -- format' % T % w % F'-e delete, create, close_write/backup/mysql/
28/05/16/backup/mysql/c
28/05/16/backup/mysql/c
28/05/16/backup/mysql/

[Root @ martin scripts] # vim inotify. sh
#! /Bin/bash
Inotify =/usr/local/inotify-tools/bin/inotifywait
Src =/backup/mysql/
$ Inotify-mrq -- format' % w % F'-e create, close_write, delete $ src \
| While read file
Do
Cd $ src &&
Rsync-az $ src -- delete tom @ sherry: mysql \
-- Password-file =/etc/rsyncd. passwd>/dev/null
Done

[Root @ martin scripts] # chmod + x inotify. sh
[Root @ martin scripts] #./inotify. sh &
[Root @ martin scripts] # jobs
[1] + Running./inotify. sh &
[Root @ martin scripts] # kill % 1
[Root @ martin scripts] # jobs
[1] + terminated./inotify. sh

# Server
[Root @ martin mysql] #/scripts/inotify. sh &
[Root @ martin mysql] # pwd
/Backup/mysql
[Root @ martin mysql] # ls
C d g kk ll
# Client
[Root @ sherry rsync-mysql] # ls
C d g kk ll
# Create a file on the server
[Root @ martin mysql] # touch pp
[Root @ martin mysql] # ls
C d g kk ll pp
# Client
[Root @ sherry rsync-mysql] # ls
C d g kk ll pp # upload completed

# Server
[Root @ martin mysql] # rm-f c d
# Cilent
[Root @ martin mysql] # ls
G kk ll pp # deletable

Add boot start
Echo '/bin/sh/scripts/inotify. sh &'>/etc/rc. local

[Root @ martin mysql] # ps-ef | grep inotify. sh
Root 126119 1 0 12:21? 00:00:00/bin/bash/scripts/inotify. sh
Root 126121 126119 0? 00:00:00/bin/bash/scripts/inotify. sh

RSync for file backup Synchronization

Monitor host files and directories using inotifywait

Using inotify + rsync for Linux File batch update

Inotify-tools + rsync real-time file synchronization installation and configuration

Complete rsync synchronization Configuration

Remote synchronization of Rsync in CentOS 6.5

Differential file comparison and extraction in Linux folders-the use of rsync

Rsync details: click here
Rsync: click here

This article permanently updates the link address: