rsync Remote Sync
The correct and effective backup scheme is an important means to ensure the security of the system and data, in the server usually combined with the scheduled task, shell script to perform local backup.
rsync (remote sync) is an open source, fast, secure and efficient offsite backup tool that mirrors and synchronizes the entire directory tree across different hosts, supports incremental backups, maintains links and permissions,
The optimized synchronization algorithm is used to perform compression before transmission. Suitable for offsite backup, image service and other applications, is a commonly used file Backup tool.
Official site: http://rsync.samba.org
Maintenance Staff: Waync Davison
Version: 3.0.9
The client responsible for initiating the rsync synchronization operation is called the initiator
The server that responds to the Rsync synchronization operation from the client is called the backup source
Download: In the downstream synchronization, the backup source is responsible for providing the original location of the document, the initiator should have read access to the file
Upload: In upstream synchronization, the backup source is responsible for providing the target location of the document, the initiator should have write permission to the file
Rsync two provenance mode:
SSH backup source
Rsync Backup Source
Configuring an SSH backup source
Advantages: Remote connection security, enhance the confidentiality of backup, easy to implement.
Configuration process:
A. Confirm the backup source folder location
B. Prepare backup operation user
Case:
A machine's site Directory/var/www/html as a backup source
User Rget do the downstream (download) backup
User rput do uplink (upload) backup
A machine for rsync server ip:192.168.200.128
b Machine for rsync client ip:192.168.200.129
Operation on a machine: 192.168.200.128:
Yum-y Install httpd
Rpm-q rsync
Rsync-3.0.6-9.el6_4.1.x86_64
Useradd Rget
echo "123456" |passwd--stdin rget
Useradd Rput
echo "123456" |passwd--stdin rput
Vi/etc/ssh/sshd_config
122 Usedns No
#110 allowusers Rget Rput
Close Usedns Accelerated SSH Login
Limit Login Users
Service sshd Restart
Adjust the/var/www/html directory permissions, so that the Rget user has Read permissions, Rput users have write permissions, it is recommended that the owner of the directory is modified to backup users, in addition to the Web service to run users to specify additional permissions,
Cd/var/www/html
mkdir Upload
Chown-r rput:rput/var/www/html/
Setfacl-r-M User:apache:rwx/var/www/html/upload
Getfacl/var/www/html/upload
ACL access control mechanism parameters detailed
Setfacl setting ACL permissions
Getfacl View ACL permissions
- R recursion
- m setting permissions
- x Individual Delete
-B Delete all
* The following two lines do not need to be executed as an understanding
Setfacl-r-b/var/www/html means delete all ACL properties
Setfacl-r-x user:apache/var/www/html/upload Delete only one item
---------------------------------------------
All new documents created later in/var/www/html/upload/, Apache is used to have RWX permissions
Setfacl-m Default:user:apache:rwx/var/www/html/upload
Getfacl/var/www/html/upload |grep Default
Configuring the Rsync Backup source
Rsync is not only used as a remote synchronous initiator (client), but also as a daemon (server side) to provide backup sources to other clients.
1. Establish/etc/rsyncd.conf configuration file
2. Create a data file for the backup account
3. Start the Rsync service process
1, establish the/etc/rsyncd.conf configuration file, the source directory/var/www/html/, backup account Backuper as an example:
[Email protected] html]# vim/etc/rsyncd.conf
UID = nobody
GID = Nobody
Use chroot = yes
Address = 192.168.200.128
Port = 873
Log file =/var/log/rsyncd.log
PID file =/var/run/rsyncd.pid
Hosts allow = 192.168.200.0/24
[Wwwroot]
Path =/var/www/html
Comment = Document Root os www.crushlinux.com
Read Only = yes
Dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z
Auth users = Backuper
Secrets file =/etc/rsyncd_users.db
UID = nobody//user name
GID = Nobody//Group name
Use chroot = yes//imprisoned in Source directory
Address = 192.168.200.128//Listening addresses
Port = 873//Listening port
Log file =/var/log/rsyncd.log//LogFile Location
PID file =/var/run/rsyncd.pid//Store process ID files location
Hosts allow = 192.168.200.0/24//allowed access to client address
[Wwwroot] //share module name
Path =/var/www/html//actual path of source directory
Comment = Document Root os www.crushlinux.com//description information
Read Only = yes/is read-only
Dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z//file types that are no longer compressed when synchronizing
Auth users = backuper//Backup authorized User
Secrets file =/etc/rsyncd_users.db//data files that hold account information
For rsync backup sources it is best to allow only read-only downstream synchronization
If you need to do upstream synchronization, it is recommended to use SSH backup source
Downstream synchronization can be anonymous, as long as the Auth users, secrets file configuration records can be removed
2, create a data file for the backup account, separated by colon, password information in the file in clear text storage, in order to avoid information leakage, need to adjust permissions
[Email protected] ~]# vim/etc/rsyncd_users.db
Backuper:pwd123
chmod 600/etc/rsyncd_users.db
The backup user Backuper also needs to have the corresponding Read permission to the/var/www/html/
ll/var/www/html/
drwxrwxr-x+ 2 rput rput 4096 January 23:06 upload
3, start the service, the operation parameter is "--daemon", turn off rsync can use KILL command
Rsync--daemon
NETSTAT-ANPT |grep rsync
Close Service
Kill $ (Cat/var/run/rsyncd.pid)
4, off-site backup of the particularity, do not need to run uninterrupted all day, configured to only the client connection when the start (to XINETD Management)
[Email protected] ~]# Vim/etc/xinetd.d/rsync
6 disable = no#将原有的yes改成no
One Server_args =--daemon#确认有--daemon option
Yum-y Install xinetd
Service xinetd Start
Using the Rsync Backup tool (at the client, initiator execution)
Local Backup: Backup source and initiator can be a machine
rsync/etc/fstab/opt/
rsync-rl/etc/fstab/boot/grub//opt/
Common formats for rsync:
rsync [Options] Original position target location
-R: Recursive mode, containing all files in directories and subdirectories
-L: The symbolic link file is still copied as a symbolic link file
-P: Retain permission flags for files
-T: Time stamp to keep files
-G: Reserved group tags for files (super users only)
-O: Preserve the main tag of the file (Superuser only)
-D: Keep device files and other special files
-A: Archive mode, merge reserved object properties, equivalent to-rlptgod
-V: Displays detailed (verbose) information for the synchronization process
-Z: Compress when transferring files (compress)
-H: Keep Hard connection files
-A: Preserve ACL attribute information
--delete: Delete a file with the destination location but not the original location
--checksum: Determines whether to skip a file based on the checksum of the object
Offsite Backup: Operations on machine B: 192.168.200.129:
To configure how the source is represented:
The rsync command needs to specify the resource location on the backup source server
Downstream backup operation: Backup source corresponds to "original location"
Upstream backup operation: Backup source corresponds to "target location"
The SSH backup source is represented as:
User name @ Host Address: Destination path, separated by a colon
Access the SSH backup source and download to the local/opt directory
Rsync-avz [Email protected]:/var/www/html//opt/
The rsync backup source is expressed as:
User name @ host Address:: Shared module Name
RSYNC://username @ host address/shared module name
Rsync-avz [Email Protected]::wwwroot/root
Rsync-avz Rsync://[email Protected]/wwwroot/root
1. downlink Synchronous SSH Backup source
Synchronize server A's/var/www/html folder with the B local/wwwroot folder (keep the file permissions property,
Soft and hard connection, ACL properties, delete redundant files in/wwwroot, transfer process to encrypt)
Mkdir-p/wwwroot
Rsync-avzh--delete [Email protected]:/var/www/html//wwwroot
ls/wwwroot/
For the same remote synchronization task, when it is executed again, the incremental update is made automatically, and the file with the same name is no longer duplicated
Operation on a machine: 192.168.200.128:
cd/var/www/html/
RM-RF index.html upload/
For i in {1..10}; Do touch $i. txt; Done
Operation on machine B: 192.168.200.129:
Rsync-avzh--delete [Email protected]:/var/www/html//wwwroot
ls/wwwroot/
2. downlink Synchronous rsync Backup Source
When the backup source is an rsync server, as long as the original location in the synchronization operation is modified to canonical format, the other operations are basically the same
Download the wwwroot from the backup source server to the local/myweb directory
Operation on machine B: 192.168.200.129:
Mkdir/myweb
Rsync-avzh--delete [Email protected]::wwwroot/myweb
Ls/myweb
3. Upstream Synchronous SSH backup source
The files in the client are uploaded to the/var/www/html directory of the backup source server, because the user is Rput is not the root user, so the-G-O option cannot be used
Cd/root
Rsync-rlvz--delete install.log [email protected]:/var/www/html
Write the rsync backup script:
Backup work in a production environment is typically repeated on a schedule, combined with Shell and Crond services to complete
1, SSH backup source without interactive authentication, because the script according to Crond time to execute, users can not be prompted to enter the password on time
Operation on machine B: 192.168.200.129: Create a key pair, send the public key file to the backup user in the a server for no interactive login
To create a key pair:
SSH-KEYGEN-T RSA
To copy a key pair:
Ssh-copy-id [email protected]
Ssh-copy-id [email protected]
Connection test:
SSH [email protected]
SSH [email protected]
Rsync-avzh--delete [Email protected]:/var/www/html//wwwroot
2, RSYNC backup source without interactive authentication, you can use export Rsync_password to save the password, when the script executes, you can automatically read the value of the variable, when needed to send to the RSYNC server for the Swallow
Export rsync_password=pwd123
Rsync-avzh--delete [Email protected]::wwwroot/wwwroot/
It is not safe to use variables to store passwords relative to the SSH source, so you can use them in your scripts and tightly control the permissions of your scripts to avoid password leaks.
VI rsync_get_wwwroot.sh
-------------------------------------
#!/bin/bash
Cmd= "/usr/bin/rsync"
Rsync_user= "Backuper"
Export rsync_password= "pwd123"
args= "-az--delete"
Src= "192.168.200.128::wwwroot"
Dst= "/wwwroot"
Mkdir-p $DST
$CMD $ARGS [email protected] $SRC $DST
------------------------------------
chmod rsync_get_wwwroot.sh
Crontab-e
* * * */root/rsync_get_wwwroot.sh
Service Crond Restart
Chkconfig Crond on
Rsync+inotify Real-time synchronization (operation on machine B)
Linux from the 2.6.13 kernel began to provide a INOTIFY notification interface to monitor the file system changes, such as file access, deletion, movement, modification and so on. With this mechanism, you can implement file alert, incremental backup, and respond to changes in directories or files in a timely manner.
Rsync+inotify can implement a triggered backup, as long as the original location of the document changes, the immediate start of an incremental backup, otherwise in a silent wait state, to avoid the fixed-cycle backup of the delay, the period is too dense and so on.
INotify mainly to do native monitoring, in the trigger backup application more suitable for upstream synchronization
The inotify mechanism provides three regulatory parameters:
Max_queued_events //Indicates monitoring event queue
Max_user_instances //maximum number of monitoring instances
Max_user_watches//maximum number of monitored files per instance
Cat/proc/sys/fs/inotify/max_queued_events //Indicates monitoring event queue (16384)
16384
Cat/proc/sys/fs/inotify/max_user_instances//Maximum number of monitoring instances (128)
128
Cat/proc/sys/fs/inotify/max_user_watches //Per instance maximum number of monitored files (8192)
8192
When you want to monitor the directory, the number of files or changes frequently, it is recommended to increase the value of these three parameters
Vi/etc/sysctl.conf
Fs.inotify.max_queued_events = 16384
Fs.inotify.max_user_instances = 1024
Fs.inotify.max_user_watches = 1048576
Sysctl-p
Installation Inotify-tools mainly provides tools such as inotifywait,inotifywatch to monitor and summarize changes.
Official site: http://inotify-tools.sourceforge.net
Tar XF inotify-tools-3.14.tar.gz
CD inotify-tools-3.14
./configure && make && make install
To monitor the site Directory/var/www/html as an example, after executing the inotifywait command, in another terminal to change the contents of the/var/www/html directory
Mkdir/var/www/html-p
Inotifywait-mrq-e modify,create,move,delete/var/www/html/
- e Specifies the events to monitor
- m means continuous monitoring
- R means recursive entire directory
- Q simplifies output information
Another terminal: Add Files in/var/www/html/directory, move files, track screen output results
mkdir/var/www/html/a
Mkdir/var/www/html/b
Mv/var/www/html/b/var/www/html/bb
rm-rf/var/www/html/a
[Email protected] ~]# inotifywait-mrq-e modify,create,move,delete/var/www/html/
fy,create,move,delete/var/www/html/
/var/www/html/create,isdir A
/var/www/html/create,isdir b
/var/www/html/moved_from,isdir b
/var/www/html/moved_to,isdir BB
/var/www/html/delete,isdir A
Inotifywait can monitor modify (modify), create (creation), move (move), delete (delete), Attrid (property change) and other events, the results of a change immediately output, inotifywait can be used to collect system changes, and output summary changes after the run is finished.
Writing trigger synchronization scripts
Inotifywait output monitoring results, each row of records include directories, events, files, which can identify the change situation, as long as the changes detected when the Rsync uplink synchronization operation can be performed.
Note: When updates are frequent, avoid concurrent rsync backups, ignore this synchronization if the rsync process already exists, or decide whether to synchronize depending on the number of rsync processes
VI inotify_rsync.sh
#!/bin/bash
inotify_cmd= "Inotifywait-mrq-e modify,create,move,attrib,delete/var/www/html/"
rsync_cmd= "Rsync-azh--delete/var/www/html/[email protected]:/var/www/html]
$INOTIFY _cmd | While read DIRECTORY EVENT FILE
Do
If [$ (pgrep rsync | wc-l)-le 0]
Then
$RSYNC _cmd
Fi
Done
chmod +x inotify_rsync.sh
Bash inotify_rsync.sh
echo "/bin/bash/root/inotify_rsync.sh"/etc/rc.local
The script is used to test the changes of the native/var/www/html directory, and once the update is started, the rsync synchronization operation, uploaded to the server 192.168.200.128/var/www/html directory, the user authentication using SSH mode.
1. Perform an upstream synchronization using the Rsync tool
2. Run the inotify_rsync.sh script on the machine
3, in the/var/www/html directory of the machine, do create, delete, modify and other operations
4. View changes in the/var/www/html directory in the server
rsync Remote Sync