Fast transfer of Big data (TAR+LZ4+PV)

Fast transfer of Big data (tar+lz4+PV) Time: the- A- A  -: in: -Read:194Comments:0Collection:0[Point my Collection +] Tags: algorithmclassstyle src uses COM log file data if the traditional SCP remote copy, the speed is relatively slow. LZ4 compression is now used for transmission. LZ4 is a very fast lossless compression algorithm with a compression speed of 300MB in a single core/S, expandable to support multi-core CPUs. It also features a very fast decoder, speed single core achievable and beyond 1gb/S. It is generally possible to achieve RAM speed limits on multicore systems. You PV all life for pipe viewer, using it we can see the progress of the command execution. Below is the introduction of the next Lz4 and PV installation, download software: Download PV-1.1.4. tar.gz Wgethttp://sourceforge.jp/projects/sfnet_pipeviewer/downloads/pipeviewer/1.1.4/pv-1.1.4.tar.bz2/under LZ4 's package is difficult, may want to Fq:https://dl.dropboxusercontent.com/u/59565338/lz4/lz4-r108.tar.gzinstallation Ash is often simple: PV installation: [Root~]$ Tar jxvf pv-1.1.4. tar.bz2 ^C[root~]$ CD pv-1.1.4[Root PV-1.1.4]$./configure && Make &&Make installlz4 installation: [Root~]$ Tar zxvf lz4-r108.tar.gz [Root~]$ CD lz4-r108[root lz4-r108]$ make &&Make Install usage: (-c Specifies the file to be transferred, SSH-P is the specified port, the IP behind is the IP of the destination host,-XC specified to the target host under the directory, the other does not have to modify): Tar-C mysql-slave-3307|pv|lz4-b4|ssh-p10022-c Arcfour128-o"MACs [email protected]"  192.168.100.234  "lz4-d |tar-xc/data"here is the effect of uploading a library from my line: bubuko.com, Bubu buckle See it, -. 7G only need to close to 3 minutes, so far faster than the SCP speed several times, direct SCP copy discrete files, very consumption of Io, and using LZ4 fast compression, the performance of little impact, fast transfer PS: The next time to add the same computer room different network segment transmission effect and cross-room transmission effect^0^

Fast transfer of Big data (TAR+LZ4+PV)