Lz4, pigz, and gzip3
1. Compression
(1.1) Use gzip for packaging:
# Time tar-zcf tar1.tar binlog *
Real 0m48. 497 s
User 0m38. 371 s
Sys 0m2. 571 s
(1.2) Use pigz compression and set the maximum compression speed (-1)
# Time tar-cv binlog * | pigz-1-p 24-k> pigz1.tar.gz
Real 0m10. 715 s
User 0m17. 674 s
Sys 0m1. 699 s
(1.3) pigz compression, default compression ratio
# Time tar-cv binlog * | pigz-p 24-k> pigz2.tar.gz
Real 0m22. 351 s
User 0m39. 743 s
Sys 0m1. 341 s
(1.4) Use pigz compression and set the maximum compression ratio (-9)
# Time tar-cv binlog * | pigz-9-p 24-k> pigz3.tar.gz
Real 0m46. 975 s
User 1m26. 118 s
Sys 0m1. 473 s
(1.5) Use lz4 for compression and set the maximum compression speed (-1)
# Time tar-cv binlog * | lz4-1> lz41.tar.gz
Real 0m4. 969 s
User 0m2. 953 s
Sys 0m1. 199 s
(1.6) Use lz4 for compression and use the default settings
# Time tar-cv binlog * | lz4> lz42.tar.gz
Real 0m4. 447 s
User 0m2. 958 s
Sys 0m1. 185 s
(1.7) Use lz4 compression and set the maximum compression ratio (-9)
# Time tar-cv binlog * | lz4-9> lz43.tar.gz
Real 0m56. 144 s
User 0m54. 300 s
Sys 0m1. 301 s
The size of the compressed file after Packaging
127 M tar1.tar (1.1)
152 M pigz1.tar.gz (1.2)
127 M pigz2.tar.gz (1.3)
123 M pigz3.tar.gz (1.4)
216 M lz41.tar.gz (1.5)
216 M lz42.tar.gz (1.6)
163 M lz43.tar.gz (1.7)
Ii. Decompression:
(2.1) directly decompress the tar file using tar-xvf
# Time tar-xvf tar1.tar
Real 0m42. 329 s
User 0m9. 134 s
Sys 0m2. 704 s
(2.2) directly decompress the pigz package file using tar-xvf (the pigz compressed file can be directly decompressed using tar)
# Time tar-xvf pigz1.tar.gz
Real 0m42. 048 s
User 0m9. 818 s
Sys 0m2. 927 s
(2.3) Use pigz-d to decompress the pigz compressed file
# File1_pigz1.tar & pigz-k-d export file.gz & time tar-xf $ file
Real 0m56. 706 s
User 0m0. 074 s
Sys 0m2. 588 s
(2.4) files compressed by lz4 cannot be decompressed using tar-xvf.
# Time tar-xvf lz41.tar.gz
Gzip: stdin: not in gzip format
Tar: Child returned status 1
Tar: Error is not recoverable: exiting now
(2.5) Use lz4-d to decompress the lz4 compressed file
# File1_lz41.tar & lz4-d folder file.gz $ file & time tar-xf $ file
Successfully decoded 1203701760 bytes
Real 0m59. 851 s
User 0m0. 064 s
Sys 0m2. 699 s
Iii. Conclusion:
The following conclusions can be drawn from the data listed above:
1. pigz is much faster than gzip (five times on the Internet), but the CPU consumption is also large (about eight times on the Internet.
2. lz4 can achieve good compression speed, but the compressed files are slightly larger than gzip and pigz.
3. Files compressed by pigz can be decompressed directly using the tar-x command, while files compressed by lz4 cannot.
From the data implementation perspective, if the system wants to increase the compression and decompression speed, you can switch to the pigz compression mode, but it will increase the cpu consumption. Pigz is suitable for scenarios that require high compression efficiency but are not affected by high CPU consumption in a short period of time.
