Faster data transmission using tar + lz4/pigz + ssh

The previous article introduced how to extract the SCP transmission speed to the maximum extent. With this foundation, we can further use compression to accelerate the transmission speed. Use scp only, transmission rate

The previous article introduced how to extract the SCP transmission speed to the maximum extent. With this foundation, we can further use compression to accelerate the transmission speed. Use scp only, transmission rate

The previous article introduced how to extract the SCP transmission speed to the maximum extent. With this foundation, we can further use compression to accelerate the transmission speed. Only scp is used, and the transmission rate is about 90 MB at the earliest. This article uses compression to increase the transmission rate to about 250 MB/s (including the decompression process ).

Directory

1. Conclusion

2. About lz4

3. Performance Environment Description

4. Experiment Test

4.1 Analysis

1. Conclusion

Use tar + lz4 + ssh to achieve maximum transmission performance:

Time tar-c sendlog/| pv | lz4-B4 | ssh-c arcfour128 \-o "MACs umac-64@openssh.com" 10. xxx. xxx.36 "lz4-d | tar-xC/u01/backup_supu" 3.91GiB 0:00:16 [249MiB/s] real 0m16. 067 s user 0m15. 553 s sys 0m16. 821 s

249 MB/s. It is 6 times the original scp (40 MB/s). It takes about 3 hours for the original GB transmission, and now it takes only 27 minutes.

Note 1: lz4's excellent decompression performance makes him very important in this case. If you do not need to decompress the transfer, you can consider using pigz/pbiz2

NOTE 2: using pv observation, the network traffic is about 80 MB, so replacing ssh with nc does not significantly improve performance.

NOTE 3: lz4 adopts-B4 (64 KB) for compression and-B7 (4 MB) for decompression, which is the optimal value in this case.

2. About lz4

Lz4 is a compression algorithm that enables "seeing people, seeing flowers and flowers". It can be well expanded on multiple cores, and Its compression speed and compression ratio do not have much advantage (pigz ), however, his decompression speed is amazing. In this case, the lz4 decompression speed is three times that of gunzip (more comparative tests ). Because of the efficient multi-core utilization during compression, and the amazing decompression, lz4 has been used in many important occasions: the Linux 3.11 kernel implements LZ4, it can also be used to compress and decompress the kernel image HBase: Add an LZ4 compression option to HFile and so on (refer ).

Lz4 is suitable for scenarios that require frequent compression and real-time decompression.

3. Performance Environment Description

Here we use the same two host environments as the previous article: ping to get RTT is 17 ms; Use iperf to test bandwidth is 115 MB (refer to the appendix );

There are several stages in the process: disk reading, packaging, compression, transmission, decompression, unpacking, and disk storage. The speed test is as follows:

3.1 disk read and drop

Disk Read (with page cache), up to 3 GB/s; disk write is about 428 MB:

# Dd if =. /sendlog.tar of =/dev/null bs = 4096 count = 1048576 1024002 + 1 records in 1024002 + 1 records out 4194314240 bytes (4.2 GB) copied, 1.33946 s, 3.1 GB/s # dd if =/dev/zero of =. /x. zero. file bs = 4096 count = 1048576 1048576 + 0 records in 1048576 + 0 records out 4294967296 bytes (4.3 GB) copied, 10.0306 s, 428 MB/s

3.2 package and unpack

The packaging and unpacking speed is greater than 350 MB/s:

# Time tar-cf sendlog.tar./sendlog/real 0m10. 996 s # time tar-xf sendlog.tar real 0m11. 564 s

3.3 compression and decompression

Many people have made comparisons on the performance (compression, decompression, and compression ratio) of each compression tool. This article will not discuss it in detail. Here, gzip/pigz lz4 bzip is selected for comparison in this test:

| Input speed | output speed | rate | speed of decoder pigz-p 16 | 327.0 MB/s | 57.2 MB/s | 17.5% | 95 MB/s lz4 | 288.0 MB/s | 79.2 MB/s | 27.5% | 264 MB/s bzip2 | 4.9 MB/s | 0.65 MB/s | 13.1% | 25.6 MB/s

For comparison and testing of compression tools, refer:

As you can see, lz4 is slightly inferior in terms of compression ratio (compared with pigz), but it has this amazing advantage in decompression speed.