The construction of rsync service for Linux cluster construction

Rsync Service Summary
Directory
Rsync Service Summary 1
The 1th Chapter Rsync Introduction 3
1.1 What is rsync 3
1.2 Rsync features 3
1.3 Rsync Common Options 3
1.4 Rsync Three modes of Operation 4
1.4.1 Local Mode 4
1.4.2 SSH Channel mode (shell mode) 4
1.4.3 Daemon Mode 5
2nd configuration of rsync daemon mode 6
2.1 Configuration of the server side 6
2.1.1 Viewing the installation Environment 6
2.1.2 to see if there is an Rsync installation package 6
2.1.3 Add rsync user to manage local directory 6
2.1.4 Generate configuration file/etc/rsyncd.conf (the file is not present by default) 6
2.1.5 Start and check whether the start is successful 8
2.1.6 creating a shared directory and granting permissions 8
2.1.7 Creating a password file and editing 8
2.1.8 Change the password file permissions and check 8
2.1.9 Viewing the Rsync service Port 8
2.1.10 Add rsync service to boot 9
2.2 Client Configuration Step 9
2.2.1 Create a password file, just write the password in the file 9
2.2.2 Setting permissions for a password file 9
2.2.3 Building a packaged directory 9
2.3 Test 9
2.3.1 Creating a test file on the client 9
2.3.2 Test Command 9
2.4 rsync Multi-module Configuration 10
2.4.1 Editing a configuration file/etc/rsyncd.conf 10
2.4.2 Creating a directory and assigning permissions 11
2.4.3 Restarting the Rsync service 11
2.4.4 testing on the client 11
2.5 rsync Multi-User Configuration 11
2.5.1 Editing a configuration file/etc/rsyncd.conf 11
2.5.2 Creating a password file/etc/rsync.password.oldboy 12
2.5.3 testing on the client 12
3rd. Service mode exclusion and no difference synchronization 12
3.1 Service Mode exclusions 12
3.2 No difference sync 13
4th 13 Business Case List
4.1 Case 1 13
4.2 Case 2 13
5th Chapter Error Recurrence 13
5.1 Service-side error 13
5.1.1 rsync server missing shared directory 13
5.1.2 rsync Server shared directory permissions not enough 14
5.1.3 Server missing User 15
5.1.4 Client Push module more than one slash 16
5.2 Client Error 16
5.2.1 Client password file is not 16

how to speed up rsync transfers

The 1th chapter about rsync
1.1 What is rsync
? RYSNC is an open source, fast and versatile tool for simultaneous, full-scale and incremental backup of local or remote data. Rsync software for various operating platforms such as Unix/linux
? An rsync command equivalent to SCP (remote backup, but rsync can achieve incremental backup), CP, RM three commands
? The difference between the CENTOS5 series and the CENTOS6 series is that the 5 series is the advanced comparison, and then in sync, the 6 series is synchronized while comparing the differences
Features of the 1.2 rsync
? Support for copying special files, such as connection files, etc.
? can have the ability to exclude specified file or directory synchronization, which is equivalent to the Exclude function of packaging command tar
? You can keep the source file or directory permissions, time, soft connection, owner, group, etc. all attributes are unchanged (-P option)
? can achieve incremental synchronization, that is, only synchronize the changed data, so the data transfer efficiency is very high
? You can use RCP, rsh, ssh, and other means to coordinate the transfer of files (rsync itself does not encrypt data)
? Files and data can be transferred via socket (process mode) (server side and client)
? Process mode transfer with anonymous or authenticated (no system users required) for easy and secure data backup and image backup

speed up rsync
1.3 Rsync Common options
-V (--verbose): Verbose mode output, which shows the progress information at the time of transmission
-Z (--compress): Data compression on transmission,--compress-level= number, can be compressed according to the compression level
-A (--archive): archive mode, which means to transfer files recursively and keep all file attributes equal to-RTOPGDL
-R (--recursive): Subdirectories are processed in recursive mode, where all directories under the directory are treated equally
-T (--time): Keep file time information (Time attribute)
-O (--owner): Keep file owner information
-P (--perms): Keep file permissions
-G (--group): Keep file group information
-D (--devices): Keep device file information
-L (--links): Keep Soft Connection file information
1.4 Rsync three modes of operation
1.4.1 Local mode

speed up rsync large files
? Format
rsync option Source Purpose
? Instance
[Email protected] tmp]# rsync-avz/etc/hosts/tmp/
Sending incremental file list

speed up rsync many small files
Hosts

Sent 183 bytes Received bytes 428.00 bytes/sec
Total size was 296 speedup is 1.38
[[email protected] tmp]# ls
Hosts
1.4.2 SSH Channel mode (shell mode)
? Format
rsync option Source user @ Remote host: remote directory (push)
rsync option user @ Remote host: remote directory source (pull)
? Instance
################################## #推动作 ######################################
[Email protected] tmp]# rsync-avz/etc/hosts-e ' ssh-p ' [email protected]:/tmp/
The authenticity of host ' 172.16.1.31 (172.16.1.31) ' can ' t be established.
RSA key fingerprint is c2:34:59:81:a2:a7:9c:0a:23:9b:cf:1d:bb:d4:8e:ad.
Is you sure want to continue connecting (yes/no)? Yes
warning:permanently added ' 172.16.1.31 ' (RSA) to the list of known hosts.
[email protected] ' s password:
Sending incremental file list
Hosts
Sent 183 bytes Received bytes 47.56 bytes/sec
Total size was 296 speedup is 1.38
################################### #拉动作 #####################################
[Email protected] tmp]# rsync-avz-e ' ssh-p [email protected]:/tmp//tmp/
[email protected] ' s password:
Receiving incremental file list
./
Hosts
. ice-unix/

Sent Notoginseng bytes received 238 bytes 110.00 Bytes/sec
Total size was 296 speedup is 1.08
[[email protected] tmp]# ls
Hosts
1.4.3 Daemon Mode
? Format
rsync option Source user @ Host:: module--password-file= Client password configuration file (push)
rsync option user @ Host:: Module destination directory--password-file= client password configuration file (pull)
? Instance
##################################### #拉动做 ###################################
[Email protected] ~]# Rsync-avz [email protected]::backup//tmp/--password-
File=/etc/rsync.password
Receiving incremental file list
./
Stu01
......
Sent 257 bytes received 551 bytes 1616.00 Bytes/sec
Total size was 0 Speedup is 0.00
[Email protected] ~]# cd/tmp/
[[email protected] tmp]# ls
Stu01 stu02 stu03 stu04 stu05 stu06 stu07 stu08 stu09 stu10
##################################### #推动做 ###################################
[Email protected] tmp]# rsync-avz/tmp/[email protected]::backup/--password-file=/etc/rsync.password
Sending incremental file list
./
Stu01
......
. ice-unix/

Sent 517 bytes received 205 bytes 1444.00 Bytes/sec
Total size was 0 Speedup is 0.00
2nd. Configuration of rsync daemon mode
2.1 Configuration of the server side
2.1.1 Viewing the installation environment
[Email protected] ~]# cat/etc/redhat-release
CentOS Release 6.9 (Final)
[Email protected] ~]# uname-r
2.6.32-696.el6.x86_64
2.1.2 to see if there is an Rsync installation package
[Email protected] ~]# Rpm-qa rsync
Rsync-3.0.6-12.el6.x86_64
Note: If you do not have the installation package, you can install it via Yum with the following installation commands:
[email protected] ~]# Yum install-y rsync
2.1.3 Add rsync user to manage local directory
[Email protected] ~]# useradd rsync-s/sbin/nologin-m
[[email protected] ~]# ID rsync
uid=501 (rsync) gid=501 (rsync) groups=501 (rsync)
Note: This user is a virtual user and does not require a login
2.1.4 Generate configuration file/etc/rsyncd.conf (the file is not present by default)
[Email protected] ~]# vim/etc/rsyncd.conf
############################### #写入的内容如下 #################################
#rsync_config____Start
#created by Oldboy
# #rsyncd. conf start##
UID = rsync
GID = rsync
Use chroot = no
Max connections = 200
Timeout = 300
PID file =/var/run/rsyncd.pid
Lock file =/var/run/rsync.lock
Log file =/var/log/rsyncd.log
#log file =/rsync/rsyncd.log
[Backup]
Path =/backup/
Ignore errors
Read Only = False
List = False
Hosts allow = 172.16.1.0/24
Hosts Deny = 0.0.0.0/32
Auth users = Rsync_backup
Secrets file =/etc/rsync.password
#rsync_config _end
[Email protected] ~]# cat/etc/rsyncd.conf #查看
? "Configuration file etc/rsyncd.conf detailed"
#rsync_config Start
#created by Oldboy
# #rsyncd. conf start##
UID = rsync #进程对应的用户 remote command use rsync to access the shared directory
GID = rsync #用户组
Use chroot = no #和安全相关的
Max connections = #最大连接数 i.e. how many customers are allowed to connect at the same time
Timeout = #连接超时时间
PID file =/var/run/rsyncd.pid #进程对应的进程号文件
Lock file =/var/run/rsync.lock #锁文件 to keep the data consistent
Log file =/var/log/rsyncd.log #日志文件 record access and error messages
[Backup] #模块名称
Path =/backup/#服务器提供访问的目录
Ignore Errors #忽略错误
Read Only = False #忽略可读 indicates writable
List = False #不能使用ls的功能 is not listed
Hosts allow = 172.16.1.0/24 #允许哪些机器可以连
Hosts deny = 0.0.0.0/32 #禁止哪些用户可以连
Auth users = Rsync_backup #虚拟用户 independent of the virtual user outside the system
Secrets file =/etc/rsync.password #虚拟用户对应的用户和密码
#rsync_config_____End
2.1.5 Start and check whether the boot was successful
[Email protected] ~]# Rsync–daemon
[Email protected] ~]# ps-ef |grep rsync|grep-v grep
2.1.6 Create a shared directory and give permissions
[Email protected] ~]# mkdir/backup/
[Email protected] ~]# chown-r rsync.rsync/backup/
[Email protected] ~]# ls-ld/backup/
Note: If the directory does not exist, will be error, detailed errors see the following error summary
2.1.7 Create a password file and edit it
[Email protected] ~]# Vim/etc/rsync.password
[Email protected] ~]# Cat/etc/rsync.password
rsync_backup:123456
Note: The format of the password file is: User: Password
2.1.8 Change the password file permissions and check
[Email protected] ~]# chmod 600/etc/rsync.password
[Email protected] ~]# ls-l/etc/rsync.password
-RW-------1 root root, Jan 20:21/etc/rsync.password
Note: The permissions for the password file are 600
2.1.9 viewing the ports of the Rsync service
[[email protected] ~]# lsof-i: 873
COMMAND PID USER FD TYPE DEVICE size/off NODE NAME
Rsync 1201 Root 4u IPv4 10536 0t0 TCP: rsync (LISTEN)
Rsync 1201 Root 5u IPv6 10537 0t0 TCP: Rsync (LISTEN)
[Email protected] ~]# Netstat-tunlp|grep 873
TCP 0 0 0.0.0.0:873 0.0.0.0:LISTEN 1201/rsync
TCP 0 0::: 873:: :LISTEN 1201/rsync
2.1.10 Add rsync service to boot
[Email protected] ~]# echo "/usr/bin/rsync--daemon" >>/etc/rc.local
[Email protected] ~]# tail-1/etc/rc.local
/usr/bin/rsync--daemon
2.2 Client Configuration steps
2.2.1 Create a password file, just write the password in the file
[Email protected] ~]# Vim/etc/rsync.password
[Email protected] ~]# Cat/etc/rsync.password
123456
2.2.2 Setting permissions for a password file
[Email protected] ~]# chmod 600/etc/rsync.password
[Email protected] ~]# ls-l/etc/rsync.password
-RW-------1 root root 7 Mar 8 10:56/etc/rsync.password
Note: The permissions for the password file are 600
2.2.3 Creating a packaged Directory
[Email protected] ~]# mkdir/backup/
2.3 Testing
2.3.1 Creating test files on the client
[Email protected] ~]# cd/backup/
[[email protected] backup]# Touch {01..10}
[[email protected] backup]# ls
Stu01 stu02 stu03 stu04 stu05 stu06 stu07 stu08 stu09 stu10
2.3.2 Test Commands
"Method 1"
[Email protected] backup]# rsync-avz/backup/[email protected]::backup/--password-
File=/etc/rsync.password
"Method 2"
[Email protected] backup]# Rsync-avz/backup/rsync://[email protected]/backup/--
Password-file=/etc/rsync.password
2.4 rsync Multi-module configuration
2.4.1 Editing a configuration file/etc/rsyncd.conf
[Email protected] oldboy]# vim/etc/rsyncd.conf
[Email protected] oldboy]# cat/etc/rsyncd.conf
#rsync_config____Start
#created by Oldboy
# #rsyncd. conf start##
UID = rsync
GID = rsync
Use chroot = no
Max connections = 200
Timeout = 300
PID file =/var/run/rsyncd.pid
Lock file =/var/run/rsync.lock
Log file =/var/log/rsyncd.log
#log file =/rsync/rsyncd.log
Ignore errors
Read Only = False
List = False
Hosts allow = 172.16.1.0/24
#hosts deny = 0.0.0.0/32
Auth users = Rsync_backup
Secrets file =/etc/rsync.password

[Backup]
Path =/backup/
[Oldboy]
Path =/oldboy/
#rsync_config_____End
2.4.2 Create a directory and give permissions
[Email protected] ~]# mkdir/oldboy/
[Email protected] ~]# chown-r rsync.rsync/oldboy/
[Email protected] ~]# ls-ld/oldboy/
2.4.3 Restarting the Rsync service
[Email protected] ~]# Pkill rsync
[[email protected] ~]# ps-ef |grep rsync |grep-v grep
[Email protected] ~]# rsync--daemon
2.4.4 testing on the client
[Email protected] ~]# rsync-avz/backup/[email protected]::oldboy/--password-
File=/etc/rsync.password
2.5 rsync Multi-User Configuration
2.5.1 Editing a configuration file/etc/rsyncd.conf
[Email protected] ~]# vim/etc/rsyncd.conf
[Email protected] ~]# cat/etc/rsyncd.conf
#rsync_config____Start
#created by Oldboy
# #rsyncd. conf start##
UID = rsync
GID = rsync
Use chroot = no
Max connections = 200
Timeout = 300
PID file =/var/run/rsyncd.pid
Lock file =/var/run/rsync.lock
Log file =/var/log/rsyncd.log
#log file =/rsync/rsyncd.log
Ignore errors
Read Only = False
List = False
Hosts allow = 172.16.1.0/24
#hosts deny = 0.0.0.0/32
Auth users = Rsync_backup
Secrets file =/etc/rsync.password

[Backup]
Path =/backup/
[Oldboy]
Path =/oldboy/
Auth users = Test
Secrets file =/etc/rsync.password.oldboy
#rsync_config_____End
2.5.2 Create a password file/etc/rsync.password.oldboy
[Email protected] ~]# Vim/etc/rsync.password.oldboy
[Email protected] ~]# Cat/etc/rsync.password.oldboy
test:123456
2.5.3 testing on the client
[Email protected] ~]# rsync-avz/backup/[email protected]::oldboy/--password-
File=/etc/rsync.password
Note: The multi-module and multi-user configuration in the document is based on the daemon service.
3rd. Service mode exclusion and no difference synchronization
3.1 Service Mode exclusions
? Command
[Email protected] backup]# rsync-avz--exclude=a/backup/[email protected]::backup/--
Password-file=/etc/rsync.password #排除一项
[Email protected] backup]# Rsync-avz--exclude={a. g}/backup/[email protected]::backup/--password-file=/etc/rsync.password #排除连续多项
[Email protected] backup]# Rsync-avz--exclude={a,c}/backup/[email protected]::backup/--password-file=/etc/ Rsync.password #排除不连续的多项

3.2 No difference synchronization
? Command
[Email protected] backup]# rsync-avz--delete/backup/[email protected]::backup/--
Password-file=/etc/rsync.password
Note: The item is as free as possible, or it will cause a bloody massacre.

4th Enterprise Case List
4.1 Case 1
An OPS person is doing a backup of the data, resulting in a full bandwidth, which prevents users from accessing the site
4.2 Case 2
The operator of a video site uses the no-difference synchronization (The following command) when publishing video to a server on the Web, causing all previous video on the server to be emptied, leaving only the video that was pushed on the day
[[Email protected] backup] #rsync-avz--delete/backup/[email protected]::backup/--
Password-file=/etc/rsync.password

5th Chapter Error Recurrence
5.1 server-side error
5.1.1 rsync server missing shared directory
? "error instance"

? "Workaround"
simply create a shared directory on the server and give it permissions
[email  protected] ~]# mkdir/backup
[[email protected] ~]# ls-ld/backup/
drwxr-xr-x 2 root root 4096 Mar 8 10: 00/backup/
[[email protected] ~]# chown-r rsync.rsync/backup/
[[email protected] ~]# ls-ld/backup/< Br>drwxr-xr-x 2 rsync rsync 4096 Mar 8 10:00/backup/
? Test
Push on client

to view checksum on the server:

5.1.2 Rsync Server shared directory permissions not enough
? "error instance"

? workaround
Simply modify the permissions on the server-side shared directory:
[[email protected] ~]# ls-ld/backup/
Drwxr-xr-x 2 root root 4096 Mar 8 10:26/backup/
[[email protected] ~]# chown-r rsync.rsync/backup/
[[EMAIL&N Bsp;protected] ~]# ls-ld/backup/
Drwxr-xr-x 2 rsync rsync 4096 Mar 8 10:26/backup/
? Test
Push on client
[email& Nbsp;protected] mail]# rsync-avz/backup/[email protected]::backup/--password-
File=/etc/rsync.password
Sending incremental file list

Sent 101 Bytes received 8 bytes 218.00 bytes/sec
Total size was 0 speedup is 0.00

View checksum on the server
[[email p Rotected] ~]# cd/backup/
[[email protected] backup]# ls
stu01 stu02 stu03 stu04 stu05 stu06 stu07 stu08 stu09 STU10

5.1.3 server missing user
? "error instance"

? "Workaround"
[[email protected] ~]# useradd rsync-s/sbin/nologin-m
[[email protected] ~]# ID rsync
uid=501 (rsync) gid=501 (rsync) groups=501 (rsync)
"test"
Push on client
[[email protected] mail]# rsync-avz/backup/[email protected]::backup/--password-
file=/etc/ Rsync.password
Sending incremental file list

Sent 101 Bytes received 8 bytes 218.00 bytes/sec
Total size was 0 Speedup is 0.00

Viewing the checksum on the server side
[Email protected] ~]# cd/backup/
[[email protected] backup]# ls
Stu01 stu02 stu03 stu04 stu05 stu06 stu07 stu08 stu09 stu10

5.1.4 client pushes the module more than one slash
? "Error Instance"

? "Workaround"
Remove multiple slashes.
5.2 Client Error
5.2.1 Client password file is not correct
? "Error Instance"

? "Workaround"
Modify password file
? Test
To push on the client:
[Email protected] backup]# rsync-avz/backup/[email protected]::backup/--password-
File=/etc/rsync.password

To view the checksum on the server:
[Email protected] ~]# cd/backup/
[[email protected] backup]# ls
Stu01 stu02 stu03 stu04 stu05 stu06 stu07 stu08 stu09 stu10

The construction of rsync service for Linux cluster construction