Set up rsync Data Synchronization backup and rsync Data Synchronization for Small and Medium-sized Clusters

Set up rsync Data Synchronization backup and rsync Data Synchronization for Small and Medium-sized Clusters

how to speed up rsync transfers 

Important NFS Problems

1. NFS client users write NFS.

1) Why do ordinary users need to write NFS.

2) Add all_squash to exports.

What is Rsync?

speed up rsync many small files

Rsync is an open-source, fast, and versatile tool that supports full or incremental local or remote data synchronization and backup. Rsync is applicable to unix, linux, windows, and other operating systems.

Rsync Introduction

speed up rsync

Rsync is short for Remote synchronization. From the software name, we can see that Rsync has the ability to quickly copy and synchronize images and remote backup data between local and remote hosts, this function is similar to the scp command in ssh, but is superior to the scp command. scp is a full copy each time, and rsync can be an incremental copy. Of course, Rsync can also replicate full and incremental data between different partitions or directories on the local host. This is similar to the cp command, but it is also better than the cp command, cp is a full copy each time, rsync supports incremental copy.

Rsync can also be used to delete files and directories, which is equivalent to the rm command.

An rsync is equivalent to scp, cp, and rm, but better than each of them.

speed up rsync large files

Rsync features

1. Supports copying special files, such as linked files and devices.

2. You can exclude the synchronization of specified files or directories, which is equivalent to the exclusion function of the packaging command tar.

3. You can maintain the permissions, time, soft and hard links, owner, group, and other attributes of the original file or directory without changing-p.

4. incremental synchronization can be implemented, that is, only the changed data is synchronized, so the data transmission efficiency is very high.

5. You can use rcp, rsh, ssh, and other methods to transfer files (rsync itself does not encrypt data ).

6. You can transmit files and data (server and client) through socket (process mode ).

7. Support anonymous or authenticated process mode transmission, which facilitates secure data backup and mirroring.

Rsync enterprise scenarios describe data synchronization between two servers: cron + rsync

Network-wide server data backup

Rsync works with inotify in Real-time Data Synchronization Rsync.

The rsync command is a client command.

Rsync generally uses three main data transmission methods.

1. data transmission between local hosts (similar to the cp command function ).

2. Use rcp, ssh, and other channels to transmit data (similar to the scp command function ).

3. Transmit data using a daemon (socket) (an important function of rsync ).

 

Push data through ssh Channel

Source end:

Target end:

Pulling data through the ssh Channel

Source end:

Key syntax description:

1.-avz is equivalent to-vzrtopgD1, indicating that the file and directory attributes remain unchanged during synchronization.

2.-P indicates the synchronization process. You can replace it with -- progress.

3.-e 'ssh-p 22' indicates data transmission through the ssh channel.-p 22 can be omitted.

4. root@192.168.31.132:/tmp/remote host system user, address, path.

5./tmp/local path.

Rsync command synchronization Parameter options

Description of common Parameter options:

-V, -- verbose: detailed mode output, transmission progress, and other information.

-Z, -- compress: compression during transmission to improve transmission efficiency. -- compress-level = NUM can be compressed by level.

-A, -- archive: indicates that the file is transmitted recursively and all file attributes are kept, which is equal to-rtopgDl.

-R, -- recursive: recursive mode for sub-directories, that is, all directories under the directory are also transmitted, note that it is lower case r.

-T, -- times: preserve the file time information.

-O, -- owner: keep the file owner information.

-P, -- perms: Keep file permissions.

-G, -- group: Keep file group information.

-P, -- progress: displays the synchronization process and transmission progress.

-D, -- devices: Keep the device file information.

-L, -- links: retain soft links.

-E, -- rsh = COMMAND: Specifies the channel protocol used to replace the rsh shell program. For example, ssh.

-- Exclude = PATTERN: Specifies the file mode that does not need to be transferred.

-- Exclude-from = file (the directory file where the file name is located)

-- Bwlimit = PATE (speed limit) limit socket I/O bandwidth.

Production parameters:-avz or-vzrtopg.

View the rsync installation package

 

 

Start rsync

 

Create an rsync user

Create shared directory/backup

Create a password file/etc/rsync. password

Modify the/etc/rsync. password permission of the password file

Rsync server configuration steps

1. vi/etc/rsyncd. conf configuration.

2. Create rsync users and share directories/backups.

Useradd rsync-s/sbin/nologin-M

Id rsync

Mkdir/backup

Chown-R rsync/backup

3. Create a password file/etc/rsync. password.

Echo "username: password">/etc/rsync. password

Chmod 600/etc/rsync. password

4. Start the service

Rsync -- daemon

Netstat-lntup | grep rsync

Ps-ef | grep rsync | grep-v grep

5. Add to auto-start

Echo "rsync -- daemon">/etc/rc. local

Cat/etc/rc. local

Procedure for configuring the rsync Client

1. Create a password file

Echo "password">/etc/rsync. password

Chmod 600/etc/rsync. password

2. rsync

Push:

Rsync-avz/tmp/rsync_backup@192.168.31.128: backup -- password-file =/etc/rsync. password

Pull:

Rsync-avz rsync_backup@192.168.31.128: backup/tmp/-- password-file =/etc/rsync. password

Rsync-avz rsync: // The rsync_backup@192.168.31.128/backup/tmp/-- password-file =/etc/rsync. password

Security optimization:

Pkill rsync # End Process rsync

Rsync -- daemon -- address = 10.0.0.10

Process Management:

Pkill process name

Killall process name

Kill pid (process number)

Smooth Process completion:

Kill-HUP 'cat/var/run/rsyncd. Pi'

Kill-USR2 'cat/var/run/rsyncd. Pi'

Rsync server troubleshooting

1. Check whether the rsync service configuration file path is correct. The correct default path is/etc/rsyncd. conf.

2. Check whether the ip CIDR blocks allowed by host allow and host deny in the configuration file are ip CIDR blocks allowed by the client.

3. Check whether the path in the path parameter in the configuration file exists and whether the permission is correct (normally the owner and group corresponding to the UID parameter in the configuration file ).

4. Check whether the rsync service is enabled. Run ps-ef | grep rsync. Whether the port has netstat-lnt | grep 873.

5. Check whether the iptables firewall and selinux enable or disable the rsync service.

6. Check whether the password file configured by rsync on the server is 600 permission and the password file format is correct. The correct format is: Username: password. The file path corresponds to the secrect files parameter in the configuration file.

7. To push data, check whether the user in the rsyncd. conf file has the read/write permission on the directory under the module.

Rsync advantages

1. Incremental Backup synchronization.

2. Supports socket (daemon ).

3. centralized backup.

Disadvantages of rsync

1. Synchronization of a large number of small files takes a long time, and sometimes the rsync process stops.

Solution: a. Package synchronization. B. drbd (synchronous copy block of the file system ).

2. Synchronization of large files (more than 10 Gb) will be interrupted. The file is hidden before full synchronization, and the file is normal after synchronization.

No synchronization difference: -- delete (used with caution). Back up before using this parameter.

Client rsync command exclusion

Exclude a single text

Rsync-avz -- exclude = a/data1/rsync_backup@192.168.31.128: oldboy -- password-file =/etc/rsync. password

Exclude multiple files

Rsync-avz -- exclude = {a, B}/data1/rsync_backup@192.168.31.128: oldboy -- password-file =/etc/rsync. password

Rsync-avz-exclude-from = paichu. log/data1/rsync_backup@192.168.31.128: oldboy -- password-file =/etc/rsync. password

Regular backup case

 

Use scp to send to another server