rsync Remote Sync

Source: Internet
Author: User
Tags inotify rsync

rsync Remote Sync


The correct and effective backup scheme is an important means to ensure the security of the system and data, in the server usually combined with the scheduled task, shell script to perform local backup.


rsync (remote sync) is an open source, fast, secure and efficient offsite backup tool that mirrors and synchronizes the entire directory tree across different hosts, supports incremental backups, maintains links and permissions,

The optimized synchronization algorithm is used to perform compression before transmission. Suitable for offsite backup, image service and other applications, is a commonly used file Backup tool.


Official site: http://rsync.samba.org

Maintenance Staff: Waync Davison

Version: 3.0.9


The client responsible for initiating the rsync synchronization operation is called the initiator

The server that responds to the Rsync synchronization operation from the client is called the backup source


Download: In the downstream synchronization, the backup source is responsible for providing the original location of the document, the initiator should have read access to the file

Upload: In upstream synchronization, the backup source is responsible for providing the target location of the document, the initiator should have write permission to the file


Rsync two provenance mode:


SSH backup source

Rsync Backup Source




Configuring an SSH backup source


Advantages: Remote connection security, enhance the confidentiality of backup, easy to implement.


Configuration process:

A. Confirm the backup source folder location

B. Prepare backup operation user


Case:

A machine's site Directory/var/www/html as a backup source

User Rget do the downstream (download) backup

User rput do uplink (upload) backup



A machine for rsync server ip:192.168.200.128

b Machine for rsync client ip:192.168.200.129



Operation on a machine: 192.168.200.128:


Yum-y Install httpd

Rpm-q rsync


Rsync-3.0.6-9.el6_4.1.x86_64


Useradd Rget

echo "123456" |passwd--stdin rget

Useradd Rput

echo "123456" |passwd--stdin rput


Vi/etc/ssh/sshd_config

122 Usedns No

#110 allowusers Rget Rput


Close Usedns Accelerated SSH Login

Limit Login Users


Service sshd Restart


Adjust the/var/www/html directory permissions, so that the Rget user has Read permissions, Rput users have write permissions, it is recommended that the owner of the directory is modified to backup users, in addition to the Web service to run users to specify additional permissions,


Cd/var/www/html

mkdir Upload


Chown-r rput:rput/var/www/html/

Setfacl-r-M User:apache:rwx/var/www/html/upload

Getfacl/var/www/html/upload


ACL access control mechanism parameters detailed


Setfacl setting ACL permissions

Getfacl View ACL permissions

- R recursion

- m setting permissions

- x Individual Delete

-B Delete all



* The following two lines do not need to be executed as an understanding

Setfacl-r-b/var/www/html means delete all ACL properties

Setfacl-r-x user:apache/var/www/html/upload Delete only one item

---------------------------------------------


All new documents created later in/var/www/html/upload/, Apache is used to have RWX permissions

Setfacl-m Default:user:apache:rwx/var/www/html/upload

Getfacl/var/www/html/upload |grep Default






Configuring the Rsync Backup source


Rsync is not only used as a remote synchronous initiator (client), but also as a daemon (server side) to provide backup sources to other clients.

1. Establish/etc/rsyncd.conf configuration file

2. Create a data file for the backup account

3. Start the Rsync service process


1, establish the/etc/rsyncd.conf configuration file, the source directory/var/www/html/, backup account Backuper as an example:


[Email protected] html]# vim/etc/rsyncd.conf


UID = nobody

GID = Nobody

Use chroot = yes

Address = 192.168.200.128

Port = 873

Log file =/var/log/rsyncd.log

PID file =/var/run/rsyncd.pid

Hosts allow = 192.168.200.0/24

[Wwwroot]

Path =/var/www/html

Comment = Document Root os www.crushlinux.com

Read Only = yes

Dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z

Auth users = Backuper

Secrets file =/etc/rsyncd_users.db


UID = nobody//user name

GID = Nobody//Group name

Use chroot = yes//imprisoned in Source directory

Address = 192.168.200.128//Listening addresses

Port = 873//Listening port

Log file =/var/log/rsyncd.log//LogFile Location

PID file =/var/run/rsyncd.pid//Store process ID files location

Hosts allow = 192.168.200.0/24//allowed access to client address

[Wwwroot] //share module name

Path =/var/www/html//actual path of source directory

Comment = Document Root os www.crushlinux.com//description information

Read Only = yes/is read-only

Dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z//file types that are no longer compressed when synchronizing

Auth users = backuper//Backup authorized User

Secrets file =/etc/rsyncd_users.db//data files that hold account information


For rsync backup sources it is best to allow only read-only downstream synchronization

If you need to do upstream synchronization, it is recommended to use SSH backup source

Downstream synchronization can be anonymous, as long as the Auth users, secrets file configuration records can be removed

2, create a data file for the backup account, separated by colon, password information in the file in clear text storage, in order to avoid information leakage, need to adjust permissions

[Email protected] ~]# vim/etc/rsyncd_users.db

Backuper:pwd123


chmod 600/etc/rsyncd_users.db


The backup user Backuper also needs to have the corresponding Read permission to the/var/www/html/


ll/var/www/html/


drwxrwxr-x+ 2 rput rput 4096 January 23:06 upload



3, start the service, the operation parameter is "--daemon", turn off rsync can use KILL command


Rsync--daemon

NETSTAT-ANPT |grep rsync


Close Service

Kill $ (Cat/var/run/rsyncd.pid)


4, off-site backup of the particularity, do not need to run uninterrupted all day, configured to only the client connection when the start (to XINETD Management)


[Email protected] ~]# Vim/etc/xinetd.d/rsync

6 disable = no#将原有的yes改成no

One Server_args =--daemon#确认有--daemon option


Yum-y Install xinetd

Service xinetd Start




Using the Rsync Backup tool (at the client, initiator execution)


Local Backup: Backup source and initiator can be a machine


rsync/etc/fstab/opt/

rsync-rl/etc/fstab/boot/grub//opt/


Common formats for rsync:

rsync [Options] Original position target location


-R: Recursive mode, containing all files in directories and subdirectories

-L: The symbolic link file is still copied as a symbolic link file

-P: Retain permission flags for files

-T: Time stamp to keep files

-G: Reserved group tags for files (super users only)

-O: Preserve the main tag of the file (Superuser only)

-D: Keep device files and other special files


-A: Archive mode, merge reserved object properties, equivalent to-rlptgod

-V: Displays detailed (verbose) information for the synchronization process

-Z: Compress when transferring files (compress)

-H: Keep Hard connection files

-A: Preserve ACL attribute information

--delete: Delete a file with the destination location but not the original location

--checksum: Determines whether to skip a file based on the checksum of the object



Offsite Backup: Operations on machine B: 192.168.200.129:


To configure how the source is represented:

The rsync command needs to specify the resource location on the backup source server

Downstream backup operation: Backup source corresponds to "original location"

Upstream backup operation: Backup source corresponds to "target location"


The SSH backup source is represented as:


User name @ Host Address: Destination path, separated by a colon

Access the SSH backup source and download to the local/opt directory

Rsync-avz [Email protected]:/var/www/html//opt/



The rsync backup source is expressed as:


User name @ host Address:: Shared module Name

RSYNC://username @ host address/shared module name


Rsync-avz [Email Protected]::wwwroot/root

Rsync-avz Rsync://[email Protected]/wwwroot/root



1. downlink Synchronous SSH Backup source

Synchronize server A's/var/www/html folder with the B local/wwwroot folder (keep the file permissions property,

Soft and hard connection, ACL properties, delete redundant files in/wwwroot, transfer process to encrypt)


Mkdir-p/wwwroot

Rsync-avzh--delete [Email protected]:/var/www/html//wwwroot

ls/wwwroot/



For the same remote synchronization task, when it is executed again, the incremental update is made automatically, and the file with the same name is no longer duplicated


Operation on a machine: 192.168.200.128:

cd/var/www/html/

RM-RF index.html upload/

For i in {1..10}; Do touch $i. txt; Done


Operation on machine B: 192.168.200.129:

Rsync-avzh--delete [Email protected]:/var/www/html//wwwroot

ls/wwwroot/



2. downlink Synchronous rsync Backup Source

When the backup source is an rsync server, as long as the original location in the synchronization operation is modified to canonical format, the other operations are basically the same

Download the wwwroot from the backup source server to the local/myweb directory

Operation on machine B: 192.168.200.129:

Mkdir/myweb

Rsync-avzh--delete [Email protected]::wwwroot/myweb

Ls/myweb



3. Upstream Synchronous SSH backup source

The files in the client are uploaded to the/var/www/html directory of the backup source server, because the user is Rput is not the root user, so the-G-O option cannot be used


Cd/root

Rsync-rlvz--delete install.log [email protected]:/var/www/html



Write the rsync backup script:


Backup work in a production environment is typically repeated on a schedule, combined with Shell and Crond services to complete


1, SSH backup source without interactive authentication, because the script according to Crond time to execute, users can not be prompted to enter the password on time

Operation on machine B: 192.168.200.129: Create a key pair, send the public key file to the backup user in the a server for no interactive login


To create a key pair:

SSH-KEYGEN-T RSA


To copy a key pair:

Ssh-copy-id [email protected]

Ssh-copy-id [email protected]


Connection test:

SSH [email protected]

SSH [email protected]


Rsync-avzh--delete [Email protected]:/var/www/html//wwwroot



2, RSYNC backup source without interactive authentication, you can use export Rsync_password to save the password, when the script executes, you can automatically read the value of the variable, when needed to send to the RSYNC server for the Swallow


Export rsync_password=pwd123

Rsync-avzh--delete [Email protected]::wwwroot/wwwroot/



It is not safe to use variables to store passwords relative to the SSH source, so you can use them in your scripts and tightly control the permissions of your scripts to avoid password leaks.


VI rsync_get_wwwroot.sh

-------------------------------------

#!/bin/bash

Cmd= "/usr/bin/rsync"

Rsync_user= "Backuper"

Export rsync_password= "pwd123"

args= "-az--delete"

Src= "192.168.200.128::wwwroot"

Dst= "/wwwroot"

Mkdir-p $DST


$CMD $ARGS [email protected] $SRC $DST

------------------------------------


chmod rsync_get_wwwroot.sh

Crontab-e

* * * */root/rsync_get_wwwroot.sh


Service Crond Restart

Chkconfig Crond on






Rsync+inotify Real-time synchronization (operation on machine B)


Linux from the 2.6.13 kernel began to provide a INOTIFY notification interface to monitor the file system changes, such as file access, deletion, movement, modification and so on. With this mechanism, you can implement file alert, incremental backup, and respond to changes in directories or files in a timely manner.


Rsync+inotify can implement a triggered backup, as long as the original location of the document changes, the immediate start of an incremental backup, otherwise in a silent wait state, to avoid the fixed-cycle backup of the delay, the period is too dense and so on.



INotify mainly to do native monitoring, in the trigger backup application more suitable for upstream synchronization


The inotify mechanism provides three regulatory parameters:


Max_queued_events //Indicates monitoring event queue

Max_user_instances //maximum number of monitoring instances

Max_user_watches//maximum number of monitored files per instance


Cat/proc/sys/fs/inotify/max_queued_events //Indicates monitoring event queue (16384)

16384

Cat/proc/sys/fs/inotify/max_user_instances//Maximum number of monitoring instances (128)

128

Cat/proc/sys/fs/inotify/max_user_watches //Per instance maximum number of monitored files (8192)

8192


When you want to monitor the directory, the number of files or changes frequently, it is recommended to increase the value of these three parameters


Vi/etc/sysctl.conf

Fs.inotify.max_queued_events = 16384

Fs.inotify.max_user_instances = 1024

Fs.inotify.max_user_watches = 1048576


Sysctl-p




Installation Inotify-tools mainly provides tools such as inotifywait,inotifywatch to monitor and summarize changes.


Official site: http://inotify-tools.sourceforge.net


Tar XF inotify-tools-3.14.tar.gz

CD inotify-tools-3.14

./configure && make && make install


To monitor the site Directory/var/www/html as an example, after executing the inotifywait command, in another terminal to change the contents of the/var/www/html directory

Mkdir/var/www/html-p

Inotifywait-mrq-e modify,create,move,delete/var/www/html/


- e Specifies the events to monitor

- m means continuous monitoring

- R means recursive entire directory

- Q simplifies output information


Another terminal: Add Files in/var/www/html/directory, move files, track screen output results


mkdir/var/www/html/a

Mkdir/var/www/html/b

Mv/var/www/html/b/var/www/html/bb

rm-rf/var/www/html/a


[Email protected] ~]# inotifywait-mrq-e modify,create,move,delete/var/www/html/

fy,create,move,delete/var/www/html/

/var/www/html/create,isdir A

/var/www/html/create,isdir b

/var/www/html/moved_from,isdir b

/var/www/html/moved_to,isdir BB

/var/www/html/delete,isdir A


Inotifywait can monitor modify (modify), create (creation), move (move), delete (delete), Attrid (property change) and other events, the results of a change immediately output, inotifywait can be used to collect system changes, and output summary changes after the run is finished.



Writing trigger synchronization scripts


Inotifywait output monitoring results, each row of records include directories, events, files, which can identify the change situation, as long as the changes detected when the Rsync uplink synchronization operation can be performed.


Note: When updates are frequent, avoid concurrent rsync backups, ignore this synchronization if the rsync process already exists, or decide whether to synchronize depending on the number of rsync processes


VI inotify_rsync.sh


#!/bin/bash

inotify_cmd= "Inotifywait-mrq-e modify,create,move,attrib,delete/var/www/html/"

rsync_cmd= "Rsync-azh--delete/var/www/html/[email protected]:/var/www/html]

$INOTIFY _cmd | While read DIRECTORY EVENT FILE

Do

If [$ (pgrep rsync | wc-l)-le 0]

Then

$RSYNC _cmd

Fi

Done



chmod +x inotify_rsync.sh

Bash inotify_rsync.sh

echo "/bin/bash/root/inotify_rsync.sh"/etc/rc.local


The script is used to test the changes of the native/var/www/html directory, and once the update is started, the rsync synchronization operation, uploaded to the server 192.168.200.128/var/www/html directory, the user authentication using SSH mode.


1. Perform an upstream synchronization using the Rsync tool

2. Run the inotify_rsync.sh script on the machine

3, in the/var/www/html directory of the machine, do create, delete, modify and other operations

4. View changes in the/var/www/html directory in the server


rsync Remote Sync

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.