Amazon EC2 servers use Rsync + Inotify for Real-Time Synchronization

Amazon EC2 servers use Rsync + Inotify for Real-Time Synchronization

Background:

Data Synchronization is required between two Amazon servers. data can be automatically synchronized to other servers as long as one server is transferred.

Install Rsync:

CentOS 6.7 comes with Rsync automatically and does not require installation.

# Instance analysis
Assume there are two servers: A and B. A is the master web server (155.28.81.0) and B is the slave server (155.28.82.0 ). We want to back up/home/test/of server A to the/home/test/directory of server B.

# Server A configuration
#### Server A compilation and Installation
Rsync compilation and installation are very simple. You only need to perform the following steps:

[Root @ www ~] # Cd/usr/local/src/
[Root @ www src] # wget http://rsync.samba.org/ftp/rsync/src/rsync-3.0.9.tar.gz
[Root @ www src] # tar zxvf rsync-3.0.9.tar.gz
[Root @ www src] # cd rsync-3.0.9
[Root @ www rsync-3.0.9] #./configure -- prefix =/usr/local/rsync/
The [root @ www rsync-3.0.9] # make
[Root @ www rsync-3.0.9] # make install but note that rsync must be installed on server A and server B, where server A runs rsync in server mode (passive, on B, rsync is run in client mode (active. In this way, the rsync daemon is run on web server A, and the client program is regularly run on B to back up the content to be backed up on web server.

#### Create a user and password authentication File

[Root @ www rsync-3.0.9] # echo "backup: bk_passwd">/usr/local/rsync/rsyncd. passwd remember that the password file created on the server contains the user name and password. The password file created on the client only has the password and no user name.

#### Set the permission to read-only

[Root @ www rsync-3.0.9] # cd/usr/local/rsync
[Root @ www rsync] # chmod 600 rsyncd. passwd otherwise, an error may be reported:
@ ERROR: auth failed on module ***
Rsync error: error starting client-server protocol (code 5) at main. c (1503)

#### Create an rsync configuration file

[Root @ www rsync] # vi/usr/local/rsync/rsyncd. conf
Uid = root
Gid = root
Use chroot = no
Max connections = 4
Strict modes = yes
Hosts allow = 121.42.46.213 # multiple hosts can be empty.
Port = 873
Pid file =/var/run/rsyncd. pid
Lock file =/var/run/rsync. lock
Log file =/var/log/rsyncd. log

[Test]
Path =/home/test
Ignore errors
Read only = true
List = false
Auth users = backup
Secrets file =/usr/local/rsync/rsyncd. passwd ### start the rsync server as a daemon

[Root @ www rsync] # rsync -- daemon -- config =/usr/local/rsync/rsyncd. the default service port of confrsync is 873. The server receives anonymous or authenticated backup requests from the customer on this port.

#### If you want to set the service to self-start, you can add rc. local

Edit/etc/rc. d/rc. local and add:

/Usr/local/rsync/bin/rsync -- daemon -- config =/etc/rsyncd. conf

# Client B Configuration

Compilation and installation are the same as above. Common Errors occur in server B.

#### Create A password authentication file to access server

[Root @ www rsync] # echo "bk_passwd">/usr/local/rsync. passwd ### set the permission to read-only

[Root @ www rsync] # chmod 0600 rsync. passwd #### after rsync is installed, run the following command to synchronize backups:

[Root @ www rsync] # rsync-vzrtopg -- delete -- progress -- password-file =/usr/local/rsync. passwd backup@115.28.81.0: test/home/test where address backup@115.28.81.0: test, backup for server A user, 115.28.81.0 for server a ip address or domain name, test for server A configuration module.

In the preceding command line-In vzrtopg, v is verbose, z is compression, r is recursive, and topg is a parameter that maintains the original file attributes such as owner and time, -- progress indicates that the detailed progress is displayed. -- delete indicates that if the server deletes the file, the client also deletes the file to ensure true consistency. -- Password-file =/usr/local/rsync. passwd is used to specify the password file, so that you can use it in the script without entering the verification password interactively. Note that the permission attribute of the password file must be set to only root readable.

Store the backup content in the/home/test/directory of the backup machine.

#### Install inotify

[Root @ www rsync] # cd/usr/local/src/
[Root @ www src] # wget http://cloud.github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz
[Root @ www src] # tar zxvf inotify-tools-3.14.tar.gz
[Root @ www src] # cd inotify-tools-3.14
[Root @ www inotify-tools-3.14] #./configure -- prefix =/usr/local/inotify
The [root @ www inotify-tools-3.14] # make
[Root @ www inotify-tools-3.14] # make install

Rsync synchronization script:

#! /Bin/bash
# Variables
Current_date = $ (date + % Y % m % d _ % H % M % S)
Source_path =/home/www/
Log_file =/usr/local/rsync/rsync_client.log

# Rsync configuration
Rsync_server = 115.28.81.0
Rsync_module = www
Rsync_user = backup
Rsync_pwd =/usr/local/rsync. passwd
INOTIFY_EXCLUDE = '(. */* \. log |. */* \. swp) $'
INOTIFY_EXCLUDE_LIST = '/usr/local/inotify/inotify_exclude.lst'
RSYNC_EXCLUDE = '/etc/rsync_exclude.list'

# Rsync client pwd check
If [! -E $ {rsync_pwd}]; then
Echo-e "rsync client passwod file $ {rsync_pwd} does not exist! "
Exit 0
Fi

# Inotify_function
# This function is used to monitor folder (/home/www) files, but exclude subfolders (storage, bootstrape/cache ).
Inotify_fun (){
/Usr/local/inotify/bin/inotifywait-mrq -- timefmt '% Y/% m/% d-% H: % M: % s' -- format' % T % w % F '\
-- Exclude $ {INOTIFY_EXCLUDE} -- fromfile $ {INOTIFY_EXCLUDE_LIST}-e modify, delete, create, move, attrib $ {source_path }\
| While read file
Do
/Usr/bin/rsync-auvrtzopgP -- exclude-from =$ {progress} -- progress -- bwlimit = 500 -- password-file =$ {rsync_pwd }$ {source_path }$ {rsync_user} @ $ {rsync_server }: :$ {rsync_module}
Done
}
# Inotify log
Inotify_fun >>$ {log_file} 2> & 1

Here, INOTIFY_EXCLUDE_LIST = '/usr/local/inotify/inotify_exclude.lst' is as follows:
[Ec2-user @ ip-172-31-7-248 rsync] $ cat/usr/local/inotify/inotify_exclude.lst
/Home/www
@/Home/www/www.xxx.com/storage
@/Home/www/www.xxx.com/bootstrap/cache
@/Home/www/xxx.com/storage

RSYNC_EXCLUDE = '/etc/rsync_exclude.list'

[Ec2-user @ ip-172-31-7-248 rsync] $ cat/etc/rsync_exclude.list
Www.xxx.com/storage
Www.xxx.com/bootstrap/cache
Xxx.com/storage

RSync for file backup Synchronization

Monitor host files and directories using inotifywait

Using inotify + rsync for Linux File batch update

Inotify-tools + rsync real-time file synchronization installation and configuration

Complete rsync synchronization Configuration

Remote synchronization of Rsync in CentOS 6.5

Rsync details: click here
Rsync: click here

This article permanently updates the link address: