rsync Remote Sync

rsync Remote Sync

The correct and effective backup scheme is an important means to ensure the security of the system and data, in the server usually combined with the scheduled task, shell script to perform local backup.

rsync (remote sync) is an open source, fast, secure and efficient offsite backup tool that mirrors and synchronizes the entire directory tree across different hosts, supports incremental backups, maintains links and permissions,

The optimized synchronization algorithm is used to perform compression before transmission. Suitable for offsite backup, image service and other applications, is a commonly used file Backup tool.

Official site: http://rsync.samba.org

Maintenance Staff: Waync Davison

Version: 3.0.9

The client responsible for initiating the rsync synchronization operation is called the initiator

The server that responds to the Rsync synchronization operation from the client is called the backup source

Download: In the downstream synchronization, the backup source is responsible for providing the original location of the document, the initiator should have read access to the file

Upload: In upstream synchronization, the backup source is responsible for providing the target location of the document, the initiator should have write permission to the file

Rsync two provenance mode:

SSH backup source

Rsync Backup Source

Configuring an SSH backup source

Advantages: Remote connection security, enhance the confidentiality of backup, easy to implement.

Configuration process:

A. Confirm the backup source folder location

B. Prepare backup operation user

Case:

A machine's site Directory/var/www/html as a backup source

User Rget do the downstream (download) backup

User rput do uplink (upload) backup

A machine for rsync server ip:192.168.200.128

b Machine for rsync client ip:192.168.200.129

Operation on a machine: 192.168.200.128:

Yum-y Install httpd

Rpm-q rsync

Rsync-3.0.6-9.el6_4.1.x86_64

Useradd Rget

echo "123456" |passwd--stdin rget

Useradd Rput

echo "123456" |passwd--stdin rput

Vi/etc/ssh/sshd_config

122 Usedns No

#110 allowusers Rget Rput

Close Usedns Accelerated SSH Login

Limit Login Users

Service sshd Restart

Adjust the/var/www/html directory permissions, so that the Rget user has Read permissions, Rput users have write permissions, it is recommended that the owner of the directory is modified to backup users, in addition to the Web service to run users to specify additional permissions,

Cd/var/www/html

mkdir Upload

Chown-r rput:rput/var/www/html/

Setfacl-r-M User:apache:rwx/var/www/html/upload

Getfacl/var/www/html/upload

ACL access control mechanism parameters detailed

Setfacl setting ACL permissions

Getfacl View ACL permissions

- R recursion

- m setting permissions

- x Individual Delete

-B Delete all

* The following two lines do not need to be executed as an understanding

Setfacl-r-b/var/www/html means delete all ACL properties

Setfacl-r-x user:apache/var/www/html/upload Delete only one item

---------------------------------------------

All new documents created later in/var/www/html/upload/, Apache is used to have RWX permissions

Setfacl-m Default:user:apache:rwx/var/www/html/upload

Getfacl/var/www/html/upload |grep Default

Configuring the Rsync Backup source

Rsync is not only used as a remote synchronous initiator (client), but also as a daemon (server side) to provide backup sources to other clients.

1. Establish/etc/rsyncd.conf configuration file

2. Create a data file for the backup account

3. Start the Rsync service process

1, establish the/etc/rsyncd.conf configuration file, the source directory/var/www/html/, backup account Backuper as an example:

[Email protected] html]# vim/etc/rsyncd.conf

UID = nobody

GID = Nobody

Use chroot = yes

Address = 192.168.200.128

Port = 873

Log file =/var/log/rsyncd.log

PID file =/var/run/rsyncd.pid

Hosts allow = 192.168.200.0/24

[Wwwroot]

Path =/var/www/html

Comment = Document Root os www.crushlinux.com

Read Only = yes

Dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z

Auth users = Backuper

Secrets file =/etc/rsyncd_users.db

UID = nobody//user name

GID = Nobody//Group name

Use chroot = yes//imprisoned in Source directory

Address = 192.168.200.128//Listening addresses

Port = 873//Listening port

Log file =/var/log/rsyncd.log//LogFile Location

PID file =/var/run/rsyncd.pid//Store process ID files location

Hosts allow = 192.168.200.0/24//allowed access to client address

[Wwwroot] //share module name

Path =/var/www/html//actual path of source directory

Comment = Document Root os www.crushlinux.com//description information

Read Only = yes/is read-only

Dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z//file types that are no longer compressed when synchronizing

Auth users = backuper//Backup authorized User

Secrets file =/etc/rsyncd_users.db//data files that hold account information

For rsync backup sources it is best to allow only read-only downstream synchronization

If you need to do upstream synchronization, it is recommended to use SSH backup source

Downstream synchronization can be anonymous, as long as the Auth users, secrets file configuration records can be removed

2, create a data file for the backup account, separated by colon, password information in the file in clear text storage, in order to avoid information leakage, need to adjust permissions

[Email protected] ~]# vim/etc/rsyncd_users.db

Backuper:pwd123

chmod 600/etc/rsyncd_users.db

The backup user Backuper also needs to have the corresponding Read permission to the/var/www/html/

ll/var/www/html/

drwxrwxr-x+ 2 rput rput 4096 January 23:06 upload

3, start the service, the operation parameter is "--daemon", turn off rsync can use KILL command

Rsync--daemon

NETSTAT-ANPT |grep rsync

Close Service

Kill $ (Cat/var/run/rsyncd.pid)

4, off-site backup of the particularity, do not need to run uninterrupted all day, configured to only the client connection when the start (to XINETD Management)

[Email protected] ~]# Vim/etc/xinetd.d/rsync

6 disable = no#将原有的yes改成no

One Server_args =--daemon#确认有--daemon option

Yum-y Install xinetd

Service xinetd Start

Using the Rsync Backup tool (at the client, initiator execution)

Local Backup: Backup source and initiator can be a machine

rsync/etc/fstab/opt/

rsync-rl/etc/fstab/boot/grub//opt/

Common formats for rsync:

rsync [Options] Original position target location

-R: Recursive mode, containing all files in directories and subdirectories

-L: The symbolic link file is still copied as a symbolic link file

-P: Retain permission flags for files

-T: Time stamp to keep files

-G: Reserved group tags for files (super users only)

-O: Preserve the main tag of the file (Superuser only)

-D: Keep device files and other special files

-A: Archive mode, merge reserved object properties, equivalent to-rlptgod

-V: Displays detailed (verbose) information for the synchronization process

-Z: Compress when transferring files (compress)

-H: Keep Hard connection files

-A: Preserve ACL attribute information

--delete: Delete a file with the destination location but not the original location

--checksum: Determines whether to skip a file based on the checksum of the object

Offsite Backup: Operations on machine B: 192.168.200.129:

To configure how the source is represented:

The rsync command needs to specify the resource location on the backup source server

Downstream backup operation: Backup source corresponds to "original location"

Upstream backup operation: Backup source corresponds to "target location"

The SSH backup source is represented as:

User name @ Host Address: Destination path, separated by a colon

Access the SSH backup source and download to the local/opt directory

Rsync-avz [Email protected]:/var/www/html//opt/

The rsync backup source is expressed as:

User name @ host Address:: Shared module Name

RSYNC://username @ host address/shared module name

Rsync-avz [Email Protected]::wwwroot/root

Rsync-avz Rsync://[email Protected]/wwwroot/root

1. downlink Synchronous SSH Backup source

Synchronize server A's/var/www/html folder with the B local/wwwroot folder (keep the file permissions property,

Soft and hard connection, ACL properties, delete redundant files in/wwwroot, transfer process to encrypt)

Mkdir-p/wwwroot

Rsync-avzh--delete [Email protected]:/var/www/html//wwwroot

ls/wwwroot/

For the same remote synchronization task, when it is executed again, the incremental update is made automatically, and the file with the same name is no longer duplicated

Operation on a machine: 192.168.200.128:

cd/var/www/html/

RM-RF index.html upload/

For i in {1..10}; Do touch $i. txt; Done

Operation on machine B: 192.168.200.129:

Rsync-avzh--delete [Email protected]:/var/www/html//wwwroot

ls/wwwroot/

2. downlink Synchronous rsync Backup Source

When the backup source is an rsync server, as long as the original location in the synchronization operation is modified to canonical format, the other operations are basically the same

Download the wwwroot from the backup source server to the local/myweb directory

Operation on machine B: 192.168.200.129:

Mkdir/myweb

Rsync-avzh--delete [Email protected]::wwwroot/myweb

Ls/myweb

3. Upstream Synchronous SSH backup source

The files in the client are uploaded to the/var/www/html directory of the backup source server, because the user is Rput is not the root user, so the-G-O option cannot be used

Cd/root

Rsync-rlvz--delete install.log [email protected]:/var/www/html

Write the rsync backup script:

Backup work in a production environment is typically repeated on a schedule, combined with Shell and Crond services to complete

1, SSH backup source without interactive authentication, because the script according to Crond time to execute, users can not be prompted to enter the password on time

Operation on machine B: 192.168.200.129: Create a key pair, send the public key file to the backup user in the a server for no interactive login

To create a key pair:

SSH-KEYGEN-T RSA

To copy a key pair:

Ssh-copy-id [email protected]

Ssh-copy-id [email protected]

Connection test:

SSH [email protected]

SSH [email protected]

Rsync-avzh--delete [Email protected]:/var/www/html//wwwroot

2, RSYNC backup source without interactive authentication, you can use export Rsync_password to save the password, when the script executes, you can automatically read the value of the variable, when needed to send to the RSYNC server for the Swallow

Export rsync_password=pwd123

Rsync-avzh--delete [Email protected]::wwwroot/wwwroot/

It is not safe to use variables to store passwords relative to the SSH source, so you can use them in your scripts and tightly control the permissions of your scripts to avoid password leaks.

VI rsync_get_wwwroot.sh

-------------------------------------

#!/bin/bash

Cmd= "/usr/bin/rsync"

Rsync_user= "Backuper"

Export rsync_password= "pwd123"

args= "-az--delete"

Src= "192.168.200.128::wwwroot"

Dst= "/wwwroot"

Mkdir-p $DST

$CMD $ARGS [email protected] $SRC $DST

------------------------------------

chmod rsync_get_wwwroot.sh

Crontab-e

* * * */root/rsync_get_wwwroot.sh

Service Crond Restart

Chkconfig Crond on

Rsync+inotify Real-time synchronization (operation on machine B)

Linux from the 2.6.13 kernel began to provide a INOTIFY notification interface to monitor the file system changes, such as file access, deletion, movement, modification and so on. With this mechanism, you can implement file alert, incremental backup, and respond to changes in directories or files in a timely manner.

Rsync+inotify can implement a triggered backup, as long as the original location of the document changes, the immediate start of an incremental backup, otherwise in a silent wait state, to avoid the fixed-cycle backup of the delay, the period is too dense and so on.

INotify mainly to do native monitoring, in the trigger backup application more suitable for upstream synchronization

The inotify mechanism provides three regulatory parameters:

Max_queued_events //Indicates monitoring event queue

Max_user_instances //maximum number of monitoring instances

Max_user_watches//maximum number of monitored files per instance

Cat/proc/sys/fs/inotify/max_queued_events //Indicates monitoring event queue (16384)

16384

Cat/proc/sys/fs/inotify/max_user_instances//Maximum number of monitoring instances (128)

128

Cat/proc/sys/fs/inotify/max_user_watches //Per instance maximum number of monitored files (8192)

8192

When you want to monitor the directory, the number of files or changes frequently, it is recommended to increase the value of these three parameters

Vi/etc/sysctl.conf

Fs.inotify.max_queued_events = 16384

Fs.inotify.max_user_instances = 1024

Fs.inotify.max_user_watches = 1048576

Sysctl-p

Installation Inotify-tools mainly provides tools such as inotifywait,inotifywatch to monitor and summarize changes.

Official site: http://inotify-tools.sourceforge.net

Tar XF inotify-tools-3.14.tar.gz

CD inotify-tools-3.14

./configure && make && make install

To monitor the site Directory/var/www/html as an example, after executing the inotifywait command, in another terminal to change the contents of the/var/www/html directory

Mkdir/var/www/html-p

Inotifywait-mrq-e modify,create,move,delete/var/www/html/

- e Specifies the events to monitor

- m means continuous monitoring

- R means recursive entire directory

- Q simplifies output information

Another terminal: Add Files in/var/www/html/directory, move files, track screen output results

mkdir/var/www/html/a

Mkdir/var/www/html/b

Mv/var/www/html/b/var/www/html/bb

rm-rf/var/www/html/a

[Email protected] ~]# inotifywait-mrq-e modify,create,move,delete/var/www/html/

fy,create,move,delete/var/www/html/

/var/www/html/create,isdir A

/var/www/html/create,isdir b

/var/www/html/moved_from,isdir b

/var/www/html/moved_to,isdir BB

/var/www/html/delete,isdir A

Inotifywait can monitor modify (modify), create (creation), move (move), delete (delete), Attrid (property change) and other events, the results of a change immediately output, inotifywait can be used to collect system changes, and output summary changes after the run is finished.

Writing trigger synchronization scripts

Inotifywait output monitoring results, each row of records include directories, events, files, which can identify the change situation, as long as the changes detected when the Rsync uplink synchronization operation can be performed.

Note: When updates are frequent, avoid concurrent rsync backups, ignore this synchronization if the rsync process already exists, or decide whether to synchronize depending on the number of rsync processes

VI inotify_rsync.sh

#!/bin/bash

inotify_cmd= "Inotifywait-mrq-e modify,create,move,attrib,delete/var/www/html/"

rsync_cmd= "Rsync-azh--delete/var/www/html/[email protected]:/var/www/html]

$INOTIFY _cmd | While read DIRECTORY EVENT FILE

Do

If [$ (pgrep rsync | wc-l)-le 0]

Then

$RSYNC _cmd

Fi

Done

chmod +x inotify_rsync.sh

Bash inotify_rsync.sh

echo "/bin/bash/root/inotify_rsync.sh"/etc/rc.local

The script is used to test the changes of the native/var/www/html directory, and once the update is started, the rsync synchronization operation, uploaded to the server 192.168.200.128/var/www/html directory, the user authentication using SSH mode.

1. Perform an upstream synchronization using the Rsync tool

2. Run the inotify_rsync.sh script on the machine

3, in the/var/www/html directory of the machine, do create, delete, modify and other operations

4. View changes in the/var/www/html directory in the server

rsync Remote Sync