Snappy
Snappy is a compression library used by many projects in Google's in-house production environment, including Bigtable,mapreduce and RPC. Google says the algorithm library has been tuned to the target, not for compression ratios or compatibility with other similar tools. On Intel Core i7 processors, the ability to process data streams on a single core reaches 250m/s-500m/s. The snappy is also optimized for the 64-bit x86 processor, which achieves at least 250mb/s compression performance and 500mb/s decompression performance at the Intel Core i7 processor single cores. Snappy has a compression rate of 1.5-1.7 for plain text, 2-4 for HTML, and 1.0 for JPEG, PNG, and other compressed data. Google strongly touts snappy's robustness, calling it "a design that will not crash even in the face of damage or malicious input," and is stable in Google's production environment, which is tested by petabytes of data compression.
Official website: http://code.google.com/p/snappy/
Fastlz
FASTLZ is an efficient lightweight compression solution Engross, whose official test data is shown in the following table:
1GB Text data test:
Official website: http://www.quicklz.com/
Lzo/minilzo
Lzo is an open source lossless compression C language library, the advantage is that compression and decompression compared to the rapid use of memory small and other characteristics (network transmission hope is the compression and decompression speed is relatively fast, the compression rate is not very high), it provides a relatively full Lzo library and a compact version of the Minilzo library, the online test data are as follows:
Test using BMP and text files on X86 Linux virtual machines (single core 256M memory, Debian 6.0 OS).
Test file |
Original size |
Size after compression |
Compression ratio |
Compression time |
Decompression time |
1.bmp |
5292054 |
159395 |
3.01% |
9.174ms |
23.037ms |
2.bmp |
6912056 |
33806 |
0.489% |
8.33ms |
36.17ms |
3.bmp |
6220856 |
5101891 |
82% |
25.78ms |
28.43ms |
Lzo.tar |
6645760 |
2457890 |
36.98% |
34.68ms |
38.62ms |
Kdoc.tar |
16660480 |
6987402 |
41.93% |
102.86ms |
108.2ms |
Kinc.tar |
18257920 |
5684927 |
31.13% |
106.87ms |
113.86ms |
Official website: http://www.oberhumer.com/opensource/lzo/
A comparison from the Hbase:the definitive guide:
Algorithm |
% remaining |
Encoding |
Decoding |
Gzip |
13.4% |
MB/s |
118 MB/s |
LZO |
20.5% |
135 MB/s |
410 MB/s |
Zippy/snappy |
22.2% |
172 MB/s |
409 MB/s |
Performance comparison of several common fast lossless compression algorithms