Use TAR+LZ4/PIGZ+SSH for faster data transfer

Using tar+lz4/pigz+faster data transfer with SSH -- One-8|Ten: 41 Category: Linux,mysql |The previous article describes how to maximize the transfer speed of the SCP, with this foundation, you can further use compression to speed up the transmission. Using only SCP, the fastest transfer rate is about 90MB, this article will increase the fastest transfer rate to about 250MB by compressing/s (including the decompression process). directory [hide]1. Conclusion2. About Lz43. Performance Environment Description3.1disk read and drop disks3.2Package , unpacking3.3compression, decompression3.4Transmission3.5Overall Process4. Experimental testing4.1Analysis5. LZ4 parameter Test5. Why not use NC6. Can it be faster Appendix reference reading1. Conclusion Using tar+lz4+SSH's way to get the maximum transfer performance: Time Tar-C sendlog/|pv|lz4-b4|ssh-c arcfour128 \ o"MACs [email protected]" Ten. xxx.xxx. $ "lz4-d |tar-xc/u01/backup_supu" 3.91GiB0:xx: -[249mib/S] Real 0m16.067s user 0m15.553s sys 0M16.821S249MB/s, duly completed. is the most primitive SCP (40mb/s) of 6 times times, the original 400GB transmission takes about 3 hours, now only takes 27 minutes. Note 1:LZ4 's excellent performance in decompression has made him very important in this case. If you do not need to decompress the transfer, you can consider using the Pigz/PBIZ2 Note 2: With PV observation, network traffic is about 80MB, so using NC to replace SSH does not have a noticeable performance boost note 3:LZ4 compression use-B4 (64KB block size), unzip the use-B7 (4MB block size), is the test optimal value of this case2. About Lz4lz4 is a let"people see love, flowers bloom"compression algorithm, can be well extended on multicore, compression speed and compression ratio is not much advantage (Pigz), but his decompression speed is very alarming, this case test LZ4 decompression is Gunzip 3 times times (more contrast test). Because of the efficient multicore use of compression, coupled with stunning decompression, LZ4 has been used in a number of important occasions: the Linux3.11 kernel implements LZ4 and can use its compression and decompression kernel image hbase:add an LZ4 compression option To hfile, etc. (reference). The LZ4 is ideal for scenes that require frequent compression and fast decompression in real time. 3. Performance environment description Use the same two host environments as in the previous article: Ping gets the RTT is 17ms, and the Iperf test bandwidth is 115MB (see appendix); The whole process has several stages: disk read--Packaging (TAR)---------------------the speed test that corresponds to the landing disk:3.1disk read and drop disk read (with page cache), to 3GB/s; disk writes approx. 428mb:# DDif=./sendlog.tar of=/dev/NULLbs=4096Count=1048576 1024002+1Recordsinch 1024002+1Records out 4194314240Bytes (4.2GB) Copied,1.33946S3.1GB/S # DDif=/dev/zero Of=./x.zero.file bs=4096Count=1048576 1048576+0Recordsinch 1048576+0Records out 4294967296Bytes (4.3GB) Copied,10.0306S428MB/s3.2packaging, unpacking and unpacking faster than 350MB/s:# Time Tar-CF Sendlog.tar./sendlog/real 0m10.996s # Time Tar-XF Sendlog.tar Real 0m11.564s3.3compression, decompression on the performance of the various compression tools (compression, decompression, compression rate) has been a lot of people have made comparisons, this article does not do a detailed discussion, here choose gzip/Pigz lz4 bzip To do this test comparison:| Input Speed | Output Speed | Rate | Speed of decoder pigz-p -|327.0mb/s | $.2mb/s |17.5% | theMB/S Lz4 |288.0mb/s | -.2mb/s |27.5% | theMB/S Bzip2 |4.9mb/s |0.65mb/s |13.1% | -.6MB/s Compression tool comparison test reference: Gzip vs Bzip2 vs LZMA vs XZ vs LZ4 vs Lzo can see that the lz4 is slightly inferior in compression (compared to PIGZ), but has this amazing advantage over the decompression speed. 3.4The transmission is described in the SCP, about 90MB the fastest transfer speed. 3.5overall process Disk read----> Packaging----> Compression------> Transfer----> Unzip-unpacking----> Landing |->tar |->gzip |->ssh |->gzip |->tar | &GT;BZIP2 |->http |->bzip |-> ... | ->nc |-&gt, ... | ->lz4 |->lz4 >400mb/s >350mb/s 79mb/s 90mb/s 72mb/s >350mb/s >400mb/s can be seen here, decompression is the biggest bottleneck, use in the decompression of the most advantageous compression tool, can let the transmission to get the maximum speed. And LZ4 is in the decompression efficiency has a huge advantage. According to the above LZ4 test, the transfer speed theoretical value is 264MB/s (transfer speed at this time is 264*27.3%=72MB), which is also the theoretical upper limit speed of this test. 4. Experimental testing using LZ4 compression transport: # Time Tar-C sendlog/|lz4|ssh-c arcfour128 \ o"MACs [email protected]" Ten. xxx.xx. $ "lz4-d |tar-xc/u01/backup_supu"Real 0m25.646s Real 0m25.911s real 0m29.019s test three times, time consuming 26s, 29s, -.6s, the average speed of transmission is: 152mb/s, network bandwidth occupies about 41.9mb/S. Compressed transport using Pigz: # time Tar-C Sendlog/|pigz-p -|ssh-c arcfour128 \ o"MACs [email protected]" Ten. xxx.xx. $ "Gzip-d|tar-xc/u01/backup_supu"rreal 0m37.030s Real 0m25.911s real 0m29.019s test three times, respectively, time consuming 37s,Panax Notoginseng.2s, *.6s, the average speed of transmission is: the.7mb/s, network bandwidth consumption approx. 19.4mb/S. Contrast found that in terms of compression pigz and lz4 is not much different, but lz4 decompression is very fast, so in this need to decompress immediately under the scene, Lz4 easy to win (bzip2 this does not need to test). 4.1analysis According to the theoretical analysis in the second section, the transmission speed should be able to 260MB, but only 152MB above/s, which shows that there is room for tuning. Continue the analysis to see where the bottleneck is: using the PV tool to observe that tar+lz4 has about 70mb/output of S: Time Tar-C SENDLOG/|LZ4|PV >/dev/NULL 1.02GiB0:xx: -[ -.8mib/s] [<=>] than the direct lz4 output, it's going to be slower.About (LZ about 79mb/s). Add one more network Ssh:time tar-C sendlog/|lz4|pv|ssh-c Arcfour128-o"MACs [email protected]" Ten. xxx.xxx. $ "Cat->/dev/null" 1.02GiB0:xx: at[ +.9mib/s] [<=>] is slower than the direct lz4 output.About (LZ about 79mb/s), the far end plus decompression and unpacking, compressed transmission speed is 41.9mb/S. Why the decline, still unclear, the author also did not think of any way to directly accelerate such a pipeline transmission, if the spectators have any suggestions, may wish to share, see whether can optimize, continue to improve speed. At this point, the transfer speed is able to 150MB/S. than the most primitive SCP (40mb/s) is about 4 times times faster, the original 400GB takes about 3 hours and now only takes 45 minutes. 5. LZ4 parameter test the previous experiment found that the LZ4 compression in the entire process was slower than expected%around, and here the difference is only one using a pipe (pipe), a direct read. Here is an attempt to increase the performance by modifying the LZ4 block size comparison: lz4-with-different-block-size Test command: forIinch' Seq4 7`; DoTime tar-c./sendlog/|lz4-b$i |PV >/dev/NULL;d One1.07GiB0:xx: One[94.4mib/s] [<=>] Real 0m11.640s user 0m10.375s sys 0m4.308s can see a significant increase in the LZ compression speed when the block size is 64KB ( to%)。 So, we're adding parameters to the LZ4-B4 to see if you can improve performance: BangIndeed, the transmission performance has increased to about 249mb/S:time Tar-C sendlog/|pv|lz4-b4|ssh-c arcfour128 \ o"MACs [email protected]" Ten. xxx.xxx. $ "lz4-d |tar-xc/u01/backup_supu" 3.91GiB0:xx: -[249mib/S] Real 0m16.067s user 0m15.553s sys 0m16.821s5. Why don't you use NC without it?!!! *NC is no faster than SSH, and NC has no advantage over SSH if compressed*NC is badly called in the script and needs to execute commands at both ends*NC requires an additional network port*NC does not encrypt6. Can it be faster in this case, the LZ4 decompression speed is 264MB/s, here to reach 249mb/s, there should be a little bit more to squeeze, but I have no recruit. Bandwidth test for Appendix Iperf: Iperf-CTen. xxx.xx. --P3999-T -------------------------------------------------------------Client Connecting toTen. xxx.xx. -, TCP Port3999TCP Window Size:16.0KByte (default) ------------------------------------------------------------ [3] LocalTen. xx.xx. $Port43838Connected withTen. xx.xx. -Port3999[ID] Interval Transfer Bandwidth [3]0.0-30.0Sec3.15GBytes903Mbits/sec Iperf-s-P3999-M------------------------------------------------------------Server listening on TCP port3999TCP Window Size:85.3KByte (default) ------------------------------------------------------------ [4] LocalTen. xx.xx. -Port3999Connected withTen. xx.xx. $Port43838[ID] Interval Transfer Bandwidth [4]0.0-30.0Sec3.15GBytes902Mbits/sec [4] MSS Size1448Bytes (MTU thebytes, Ethernet) Reference reading*[email protected] code* LZ4's Details*LZ4 streaming Format*Quick benchmark:gzip vs Bzip2 vs LZMA vs XZ vs LZ4 vs LZO* lz4:extremely Fast Compression algorithm

Use TAR+LZ4/PIGZ+SSH for faster data transfer