Make remote transmission of large files faster

I did a test on two virtual machines. If the traditional SCP is used for remote copy, the speed is relatively slow. lz4 compression is used for transmission now. LZ4 is a very fast lossless compression algorithm with a compression speed of MBS per core and scalability.

I did a test on two virtual machines. If the traditional SCP is used for remote copy, the speed is relatively slow. lz4 compression is used for transmission now. LZ4 is a fast lossless compression algorithm with a compression speed of 300 MB/S per core and scalability.

I did a test on two virtual machines.

If the traditional SCP is used for remote copying, the speed is relatively slow,

Lz4 compression is used for transmission. LZ4 is a fast lossless compression algorithm with a compression speed of 300 MB/S per core. It can be expanded to support multi-core CPUs. It also has a very fast decoder with a single-core speed that can reach and surpass 1 GB/S. Generally, it can reach the RAM speed limit on multi-core systems.

Project homepage:

Install (lz4 compression tool is installed both locally and remotely)

# Wget https://dl.dropboxusercontent.com/u/59565338/LZ4/lz4-r108.tar.gz

# Cd lz4-r108/

# Make; make install

Remote Transmission

Time tar-c aa. img | pv | lz4-B4 | ssh 192.168.17.129 "lz4-d | tar-xC/root /"

2G files are transmitted at a speed of 119 Mb/s, which takes 17 seconds.

This article is from the "hechun's technical column" blog. Be sure to keep this source