Large file transfer between servers

Preface
In managing MySQL, initializing servers, replication, and backup/restore, transferring large files across the network is a common operation.

1, the basic operation 
Compress large Files
Send to another server
Unzip large files
Verify file integrity, corruption

2, the specific operation 
(1) General steps (SCP)
Server1:
Gzip-c/backup/mysql/mytable. MYD > MyTable. Myd.gz
SCP mytable. myd.gz [email protected]:/var/lib/mysql/
Server2:
Gunzip/var/lib/mysql/mytable. Myd.gz
Characteristics:
SCP transfer speed is slow, but ensure the security of transmission, and these operations a large number of disk read and write, so that the efficiency of transmission is not high.
Note:
Scp-r/data/file [Email protected]:/data/
Scp-c/data/sda.img [Email protected]:/data/img/
#-R: Support Directory
#-C: Enable compression transfer

(2) Pipeline transfer (SSH)
Gzip-c mytable. MYD | SSH [email protected] "gunzip-c->/var/lib/mysql/mytable. MYD "
Characteristics:
Reduces the disk I/O compared to the previous one, making transmission more efficient.
Note:
# to MyTable. MyD using gzip compression, the-c parameter indicates output to stdout, which is piped
# gunzip-c-"-" means receiving a stdin from a pipe

(3) data stream redirect Transmission (NC)
Principle:
A listening port is opened in the destination server, and the source port transmits the file to the listener port of the destination server.
Server1:
Gzip-c-/var/lib/mysql/mytable. MYD |nc-q 1 Server1 12345
Server2: Turn on listening port 12345
Nc-l-P 12345 |gunzip-c->/var/lib/mysql/mytable. MYD
Note:
#-Q: The transfer completes the file, then closes and exits
#-L: Turn on listening mode
Features: Netcat can be transmitted in the process, the data will not be decrypted, and SSH to the data transmission encryption and decryption, so that the Netcat tool can increase the speed of transmission.

(4) Rsync transmission (breakpoint continuation)
rsync-av/backup/-e ssh [email protected]:/bak
#-a:archive archive mode, which means to transfer files recursively and keep all file attributes, links, etc.
Characteristics:
Support the continuation of the breakpoint, encrypted transmission of data, you can maintain the original file permissions, time, soft and hard links.

3. Summary 
The above is only for a variety of tool commands to make comparisons, but specifically which in the file transfer more efficient, need for the server's environmental considerations, analysis of disk drives, network cards, CPUs, tested experiments to obtain the most efficient method.
If: Use Vmstat-n 5 to analyze whether the disk or CPU is the bottleneck of the transmission speed. If the CPU is very high, and the network and disk is relatively rich, you can not compress the direct transmission, if the network is a bottleneck, and the CPU is well-off, you can compress and transfer files, considering the speed of the problem, can use parallel, but the load is not high, otherwise excessive concurrency will slow down.

This article is from the "Flower Fall" blog, please be sure to keep this source http://starglm.blog.51cto.com/8806294/1826822

Large file transfer between servers