Powerful synchronization tool rsync detailed configuration and various gameplay, rsync detailed configuration gameplay

Powerful synchronization tool rsync detailed configuration and various gameplay, rsync detailed configuration gameplay

Terms:
Sync: Synchronization
Async: asynchronous
Rsync: Remote Synchronization

Rsync features:

1. images can be used to save the entire directory tree and File System
2. It is easy to retain the original permissions (permission, mode), owner, group, time (modification time, modify time), soft and hard links, File acl, and file attributes.
3. high transmission efficiency. Only the changed
4. Anonymous transmission is supported to facilitate website images. verification can also be performed to enhance security.


Rsync parameters (Common-av)
-V verbo
-A archive mode: Transfers files recursively and maintains the attributes of files.
-L retain soft links
-R reserve relative path
-H retained hard links
-P,-o,-g, and-A retain permissions, owner, group, acl, and so on, but if-a is added, these include
-D equals to -- devices -- specials indicates that files of the B, c, s, and p types are supported.
Example:
Rsync-av/home // backup/-- synchronize the content in the/home directory to the/backup Directory (including hidden files)
Rsync-av/home/10.1.1.218:/backup/-- synchronize the local/home directory content to the/backup Directory of remote 218. A remote password of 10.1.1.218 is required.
Rsync-av 10.1.1.218:/backup // backup/-- synchronize the content in the remote 218/backup Directory to the local/backup directory. A remote password of 10.1.1.21 is required.
Rsync-a/test/'-e ssh-p 2222' 10.1.1.218:/test1/-- if the ssh port of the remote machine 10.1.1.218 is not the default 22, for example, it is changed to 2222, so you need to write it like this
Notes:
1. Retain the original path during synchronization, that is, synchronizing/home to/backup will become/backup/home/* And then content

-- Adding/after the original directory also affects your synchronization

There are three differences:
Rsync-av/home // backup/--/home content is directly synchronized to/backup

Rsync-av/home/backup/
Rsync-aRv/home // backup/-- the two files are/home synchronized to/backup/home /*

# Rsync-av/share/20171225/backup/-- synced to/backup/20171225 /*

# Rsync-aRv/share/20171225/backup/-- synced to/backup/share/20171225 /*

Rsync-aHAX -- delete/test // test1/-- a complete method (with the -- delete parameter added, the source file is deleted, and the target file is also deleted)

Rsync can be used as a service (daemon), and some control functions, logging functions, anonymous login or login authentication functions can be added.

First

yum -y install rsync

 

Requirement 1: do not add other parameters except for the homepage file.
Step 1
Vim/etc/rsyncd. conf -- modify the configuration file man rsyncd. conf.
Motd file =/etc/rsyncd. welcome -- welcome file, custom path

[Notes] -- the name shared to the client. The name is customized.
Path =/share/20171225/-- the actual server path to be shared. The name must be the actual path to be shared.

Vim/etc/rsyncd. welcome -- manually create your own motd file and write the custom welcome information. You can save wq.
Step 2
Systemctl restart rsyncd
Systemctl enable rsyncd
Lsof-I: 873 -- restart the service and check whether the port is in the listening status
Step 3: Check the client
Rsync-a 192.168.2.9: -- You can directly view what the server 192.168.2.9 shares.
"Hello today! @ _ @ "-- This is the welcome content written in your rsyncd. welcome.
Notes -- this is the shared name.

Or directly synchronize the shared file (note that the server IP address is followed by two colons, followed by your custom tag name, without adding a path)
Rsync-a 192.168.2.9: notes/

Requirement 2: Add Special Control Parameters
1. log to/var/log/rsyncd. log
2. The shared module must be hidden (that is, the client cannot view the module name)
3. At the same time, only one client can be connected to synchronize this module.
4. Only 192.168.2.x (custom ip address) can be allowed to synchronize this vim/etc/rsyncd. conf
Vim/etc/rsyncd. conf -- modify the configuration file and add the control parameters.

Motd file =/etc/rsyncd. welcome
Syslog facility = local0 -- change the log carrier to local0

[Notes]
Path =/share/20171215/
Log file =/var/log/rsyncd. log -- the log time uses the Greenwich Mean Time.
# Syslog facility = local0 -- no problem in time, but this parameter must be written in the global parameter and combined with rsyslog to generate logs (it should be a small bug, but this problem exists in rhel6)
List = false -- hide the list
Max connections = 1 -- control the maximum number of connections
Hosts allow = 192.168.2.X -- control the access IP Address

Restart service
Systemctl restart rsyncd

Requirement 3: implement real-time synchronization (synchronization is triggered when the source file changes)
Inotify + rsync


Under the note directory/program/inotify_soft/inotify-tools-3.13.tar.gz-software to be installed

# Tar inotify-tools-3.13.tar.gz-C/usr/src/
# Cd/usr/src/inotify-tools-3.13/
#./Configure
# Make
# Make install

-- After installation, the following two commands are generated:
/Usr/local/bin/inotifywait
/Usr/local/bin/inotifywatch

1) Real-time synchronization of one-way local directories
The/dst directory is synchronized to the/src directory as soon as there is (add, delete, modify, and attribute change) in the local/src directory.

-- Use the following script

# Vim/tmp/1.sh #! /Bin/bash/usr/local/bin/inotifywait-mrq-e modify, delete, create, attrib, move/src | while read events -- the first line indicates that dorsync-a -- delete/src/dst/echo "'date + '% F % T' is triggered when addition, deletion, and modification occur'' event $ events ">/var/log/rsync appears. log 2> & 1 -- point to log Path done

 

# Nohup sh/tmp/1.sh & -- use nohup to suspend the execution in the background. If the terminal is closed, the process will not be closed.
-- If you want To Enable Automatic startup, you can add this command to/etc/rc. local or write it as a shell Service script to enable chkconfig to automatically start upon startup.


Then perform various operations on/src to verify that the data can be synchronized to/dst in real time.

-- Check the sersync tool online after class



2) implement real-time synchronization of two-way local directories
There are also related software (such as unison) that can be used for two-way real-time synchronization.
However, we can use inotify to reverse and write another script.



========================================================== ==============================================


Requirement 4: Remote Real-Time Synchronization

Method 1: Use ssh equivalent (ssh key logon) to enable remote machine connection without a password, so it is easy to use remote monitoring or synchronization.


Synchronize the/src directory on 10.1.1.1 to the/dst directory on 10.1.1.2 in real time.
10.1.1.1/src directory 10.1.1.2/dst directory


Generate a pair of keys on 10.1.1.1-common methods for generating keys on Linux, interactive
# Ssh-keygen -- allows you to enter the password file storage path
# Ssh-copy-id-I 10.1.1.2
# Ssh-add/root/. ssh/id_rsa


Then implement remote real-time synchronization. You only need to add the remote IP address in front of the remote directory based on the original script (because ssh is equivalent, no password is required)

#! /Bin/bash/usr/local/bin/inotifywait-mrq-e modify, delete, create, attrib, move/src | while read eventsdorsync-a -- delete/src/10.1.1.2: /dst/echo "'date + '% F % t' Event events">/var/log/rsync. log 2> & 1 done

 

Method 2: Do not use the ssh key of method 1 above. Use the verify CT script to solve the password problem.

#! /Bin/bash/usr/local/bin/inotifywait-mrq-e modify, delete, create, move/src | while read eventsdosed-I '/^ 10.1.1.2/D'/root /. ssh/known_hostsexpect <EOF &>/dev/nullspawn rsync-a -- delete/src/10.1.1.2:/dst/keep CT "no )? "Send" yes \ r "Keep CT" password: "sleep 2 send" 123456 \ r "expect CT eofEOFecho" 'date + '% F % t' Event events "& >>>/var/log/rsync. logdone

 

Test: First kill all the script processes executed above, and then re-execute this script for verification.




Method 3: inotify + rsync + nfs
The idea of this method is to mount a remote directory to a local directory through the nfs service and use it like a local directory. This avoids password loss during rsync remote synchronization.
In rhel6.5, the test shows that the system fails to monitor the remote nfs Mounted directory and synchronize it to the local directory. However, the local directory operation will also be synchronized to the local directory. Cause: inotify can only monitor directories on the local machine, but cannot monitor directories on another machine)
However, you can use innotify to monitor the local directory and then synchronize it to the remote nfs mount directory.



Synchronize the/src directory on 10.1.1.1 to the/dst directory on 10.1.1.2 in real time.
10.1.1.1/src directory 10.1.1.2/dst directory

Step 1:
On 10.1.1.2
# Vim/etc/exports -- configuration file for nfs Mounting Parameters
/Dst * (rw, no_root_squash)
# Systemctl restart nfs

Step 2:
On 10.1.1.1
# Mkdir/dst/
# Mount-t nfs 10.1.1.2:/dst -- mount it. The/dst directory is the remote/dst directory on 10.1.1.2.

Step 3:
Execute the above two local directories to synchronize the same script, the test is OK

# Vim/tmp/1.sh #! /Bin/bash/usr/local/bin/inotifywait-mrq-e modify, delete, create, attrib, move/src | while read eventsdorsync-a -- delete/src/dst/echo "'date + '% F % t' event $ events" >>>/var/log /rsync. log 2> & 1 done

 

# Nohup sh/tmp/1.sh &

Step 4:
Test (process omitted), but note that the script process run in the previous example is found through ps before the test, and kill

-- Expansion: If nfs is replaced with the samba service, is it feasible to mount the remote samba shared directory to the local through the mount-t cifs command?





Method 4: Use the secrets file method of rsync and then write the script

Example:
Local synchronization end -----> remote synchronization end
10.1.1.110.1.1.2
I want to synchronize the content in the/src/directory on the local synchronization end 10.1.1.1 to the/dst/directory of the remote synchronization end 10.1.1.2 in real time.


Preparation:
Disable iptables and selinux
Host Name binding
Time Synchronization


Implementation Process
Step 1: operate on the remote synchronization terminal 10.1.1.2
# Mkdir/dst/-- create the synchronized directory for the test

# Yum install rsync

# Vim/etc/rsyncd. conf
Uid = root
Gid = root -- use the root identity to synchronize to 10.1.1.2. Otherwise, the nobody user is used by default. (In this way, you need to change the/dst/directory to 757 to have the write permission)

[Testsource]
Path =/dst/
Ignore errors
Read only = no -- Write Permission for the Service Level
Auth users = test -- the authenticated user named test (this user is not related to the System user)
Secrets file =/etc/rsyncd. secrets -- verify that the user test's user name and password are stored in the file. This file must be available on both the synchronization end and the synchronization end (so that we can use
This method does not require remote ssh password verification)
Hosts allow = 10.1.1.1 -- to further enhance security, only 10.1.1.1 synchronization is allowed


# Vim/etc/rsyncd. secrets -- create this password file. The username must be the same as the password specified by auth users above. The password is random (I will test it with 321 here)
Test: 321

# Chmod 600/etc/rsyncd. secrets -- change the permission to 600 for security purposes and only allow root to read and write data (You are required to change the password file of many other software to 600 for permissions)



# Systemctl restart rsyncd -- restart the service
# Systemctl enable rsyncd

# Netstat-ntlup | grep: 873 -- verify that port 873 is enabled
Tcp 0 0: 873: * LISTEN 50438/xinetd


Step 2: operate on the local synchronization terminal 10.1.1.1
Install the inotify software first. Refer to the process above.

# Mkdir/src/-- create a test synchronization directory
# Vim/etc/rsyncd. secrets -- create the password file, which only requires the password. You can specify
321
# Chmod 600/etc/rsyncd. secrets -- also changed to 600 permission

# Vim/tmp/rsync. sh -- script and its content (here, it is convenient to test, so it is put under/tmp, and the production environment can be put into a directory like/bin to make a command)

#!/bin/bash/usr/local/bin/inotifywait -mrq -e modify,delete,create,attrib,move /src |while read eventsdorsync -a --delete --password-file=/etc/rsyncd.secrets /src/ test@10.1.1.2::testsourcedone

 

# Chmod 755/tmp/1.sh -- Grant the execution permission to facilitate execution

# Nohup sh/tmp/1.sh & -- the execution is suspended later

# Vim/etc/rc. local -- add it to rc. local to enable it to start automatically
Nohup sh/tmp/1.sh &

The third step:
Test process omitted