Fast transfer of Big data (TAR+LZ4+PV)

If you use traditional SCP remote copy, the speed is relatively slow. LZ4 compression is now used for transmission. LZ4 is a very fast lossless compression algorithm that compresses at a single-core 300mb/s and scales to support multicore CPUs. It also has a very fast decoder, the speed single core can reach and surpass 1gb/s. It is generally possible to achieve RAM speed limits on multicore systems.

You PV all life for pipe viewer, using it we can see the progress of the command execution.

The following describes the installation of the next lz4 and PV, download software:

Download pv-1.1.4.tar.gz wget http://sourceforge.jp/projects/sfnet_pipeviewer/downloads/pipeviewer/1.1.4/pv-1.1.4.tar.bz2/

Under Lz4 's bag is difficult, may want to fq:https://dl.dropboxusercontent.com/u/59565338/lz4/lz4-r108.tar.gz

Installation ash is often simple:

PV Installation:

tar jxvf pv-1.1. 4. tar. bz2 ^~]$ cd pv-1.1. 4 [Root PV-1.1. 4  Make  Make Install

LZ4 Installation:

tar zxvf lz4-r108. Tar  ~]$ cd lz4-r108[root lz4make doinstall

Usage: (-c after specifying the file to be transferred, SSH-P is the specified port, the IP behind is the destination host IP,-XC designated to the target host under the directory, the other does not have to modify):

tar -c mysql-slave-3307  |pv|lz4-b4|  SSH -p10022-c arcfour128-o"MACs [email protected]"  192.168.  100.234  "lz4-d |tar-xc/data"

Here is the effect of uploading a library from my line:

See, 25.7G only need to close to 3 minutes, so far faster than the SCP speed several times, direct SCP copy discrete files, very consumption of Io, and using LZ4 fast compression, the performance of small impact, transmission speed

PS: The next time to add the same computer room different network segment transmission effect and cross-room transmission effect ^0^

Fast transfer of Big data (TAR+LZ4+PV)