LZO Use and introduction

Lzo Description Summary

LZO is a lossless compression library written in ANSI C language. He is able to provide very fast compression and decompression functions. Decompression does not require memory support. Even with very large compression ratios, the data that is slowly compressed can still be decompressed very quickly. LZO complies with the GNU GPL license.

Introduced

The LZO is ideal for real-time compression and decompression of data, which means he is more concerned with operating speed than compression ratio.

LZO is written in ANSI C, and the compressed data is also designed to be used across platforms.

LZO has the following features:

• The decompression speed is very fast, and very simple;
• Do not need memory support when extracting;
• The speed of compression is also good;
• Compression requires only a KiB memory support;
• Compression ratio can be adjusted according to needs, and this does not affect the efficiency of decompression, increase the compression ratio will naturally reduce the compression speed;
• Compression contains a lot of compression levels, providing many options;
• Provides compression levels that require only 8 KiB of memory support;
• Provide thread safety;
• Provide lossless compression;

Lzo provides multiple compression, and recovery decompression;

Design standards

The LZO is designed with the principle of processing speed. The decompression speed is faster than the compression speed. Ability to provide real-time decompression of any program. The decompression of lzo1x is optimized according to i386 coding.

In fact, it is the compression data structure defined by the decompression algorithm, and finally the manual test data is used to verify the structure.

Effect

On the old Pentium 133 device, we ran the test data for Calgary Corpus. The data uses the size of the KiB.

Lzoxx-n defines the algorithm name used, and N represents the compression level. With 1-9 levels of KiB memory, he mainly provides faster compression speeds. Class 99 uses the KiB memory to provide a larger compression ratio, but processing is still fast. Level 999 is an algorithm optimized by compression ratio, which is slow to compress and uses a lot of memory, which is typically used to generate pre-compressed data.

The C language version of the lzo1x-1 algorithm is 4-5 times faster than the fastest ZLIB compression level. Of course he also in compression ratio, compression time, decompression time is more convenient than lzrw1-a and lzv these algorithms.

 +------------------------------------------------------------------------+ | Algorithm Length CxB comlen%remn Bits Com k/s Dec k/s | |                                                                        ---------        ------  ---   ------  -----  ----   -------   ------- | | | |                                                                        memcpy () 224401 1 224401 100.0 8.00 60956.83 59124.58 | | | | Lzo1-1 224401 1 117362 53.1 4.25 4665.24 13341.98 | |                                                                        lzo1-99 224401 1 101560 46.7 3.73 1373.29 13823.40 | | | | Lzo1a-1 224401 1 115174 51.7 4.14 4937.83 14410.35 | |                                                                        lzo1a-99 224401 1 99958 45.5 3.64 1362.72 14734.17 | | | | Lzo1b-1 224401 1 109590 49.6 3.97 4565.53 15438.34 | | Lzo1b-2 224401 1 106235 48.4 3.88 4297.33 15492.79 | | Lzo1b-3 224401 1 104395 47.8 3.83 4018.21 15373.52 | | Lzo1b-4 224401 1 104828 47.4 3.79 3024.48 15100.11 | | Lzo1b-5 224401 1 102724 46.7 3.73 2827.82 15427.62 | | Lzo1b-6 224401 1 101210 46.0 3.68 2615.96 15325.68 | | Lzo1b-7 224401 1 101388 46.0 3.68 2430.89 15361.47 | | Lzo1b-8 224401 1 99453 45.2 3.62 2183.87 15402.77 | | Lzo1b-9 224401 1 99118 45.0 3.60 1677.06 15069.60 | | lzo1b-99 224401 1 95399 43.6 3.48 1286.87 15656.11 | |                                                                        lzo1b-999 224401 1 83934 39.1 3.13 232.40 16445.05 | | | | Lzo1c-1 224401 1 111735 50.4 4.03 4883.08 15570.91 | | Lzo1c-2 224401 1 108652 49.3 3.94 4424.24 15733.14 | | Lzo1c-3 224401 1 106810 48.7 3.89 4127.65 15645.69 | | Lzo1c-4 224401 1 105717 47.7 3.82 3007.92 15346.44 | | Lzo1c-5 224401 1 103605 47.0 3.76 2829.15 15153.88 | | Lzo1c-6 224401 1 102585 46.5 3.72 2631.37 15257.58 | | Lzo1c-7 224401 1 101937 46.2 3.70 2378.57 15492.49 | | Lzo1c-8 224401 1 100779 45.6 3.65 2171.93 15386.07 | | Lzo1c-9 224401 1 100255 45.4 3.63 1691.44 15194.68 | | lzo1c-99 224401 1 97252 44.1 3.53 1462.88 15341.37 | |                                                                        lzo1c-999 224401 1 87740 40.2 3.21 306.44 16411.94 | | | | Lzo1f-1 224401 1 113412 50.8 4.07 4755.97 16074.12 | |                                                                        lzo1f-999 224401 1 89599 40.3 3.23 280.68 16553.90 | | | | Lzo1x-1 (11) 224401 1 118810 52.6 4.21 4544.42 15879.04 | | Lzo1x-1 (12) 224401 1 113675 50.6 4.05 4411.15 15721.59 | | Lzo1x-1 224401 1 109323 49.4 3.95 4991.76 15584.89 | | Lzo1x-1 (15) 224401 1 108500 49.1 3.93 5077.50 15744.56 | |                                                                        lzo1x-999 224401 1 82854 38.0 3.04 135.77 16548.48 | | | | Lzo1y-1 224401 1 110820 49.8 3.98 4952.52 15638.82 | |                                                                        lzo1y-999 224401 1 83614 38.2 3.05 135.07 16385.40 | | | |                                                                        lzo1z-999 224401 1 83034 38.0 3.04 133.31 10553.74 | | | | lzo2a-999 224401 1 87880 40.0 3.20 301.21 8115.75 | +------------------------------------------------------------------------+

Annotations:

• CxB is the number of blocks;
• K/S is the speed at which 1KB of uncompressed data is processed per second;
• The decompression speed of the assembly will be faster;

Simple document

LZO is a block compression algorithm that compresses and decompress a block of data. The block size must be the same for compression and decompression.

LZO compresses block data into matching data (sliding dictionaries) and non-matching text sequences. LZO for long-match and long-text sequences have a special treatment, so for high-redundancy data can be very good results, so for the non-compressible data, but also to obtain better results.

When processing non-compressible data, the LZO expands the input data by 64 bytes per 1024 bytes.

I have verified the LZO program through a memory checker like Valgrind, and I have used data that is more than gigabytes, tuning various parameters and checking for potential problems. LZO There are no known bugs at this time.

Various algorithm descriptions

A lot of algorithms are implemented here, but I want to provide unrestricted backwards compatibility, so I won't cancel the existing content.

Just as the object files are relatively independent, if you use the LZO library through static linking, only a small amount of capacity (about a few KiB) is added, assuming that the module will be loaded only if the LZO function is actually used.

In March 1996, I announced the LZO1 and lzo1a in Newsgroups.comp.compression and Comp.compression.research. They dealt primarily with compatibility issues. LZO2A decompression is very slow, and does not provide a faster compression speed.

As can be seen in my experiment, LZO1B is suitable for processing large amounts of data, or for data with high redundancy. The lzo1f is suitable for handling small amounts of data and binary data. LZO1X is suitable for a variety of environments. LZO1Y and lzo1z are similar to lzo1x, and they are able to achieve a better compression ratio in some environments.

Please note that you have a lot of options.

Using the LZO Library

The basic functionality of using the LZO library is simple, regardless of the size of the data. We assume that you intend to use Lzo1x-1 's algorithm to process your data.

Compression

1 include <lzo/lzo1x.h>  2calllzo_init ()  3 via lzo1x_1_ Compress () Compress your data   4 Compile the connection LZO library  

Extract

1 include <lzo/lzo1x.h>  2calllzo_init ()  3 by Lzo1x_decompress () Extracting data   4 compiling connection LZO Library  

The examples/simple.c of the source package has a complete sample code. You can also view LZO. FAQ for more information.

Original file:

• http://www.oberhumer.com/opensource/lzo/lzodoc.php

LZO FAQ

2. I don't want to read the documentation, point to know how to add this feature to my program

   There are two examples: examples/simple.c and EXAMPLES/LZOPACK.C provide a simple way to use.

   But it's better to read the entire document, and it will help you get it done.

4. Can you explain the meanings of the various algorithm names?

   We use lzo1x to illustrate:

   Algorithm Name: lzo1x

   Algorithm Category: LZO1

   It then provides a lot of compression levels:

        lzo1x-999        !----------algorithm classification         !---------algorithm type           !!! -----compression Level (1-9, 99, 999)

        Lzo1x-1 (one)        !----------algorithm classification         !---------algorithm type           !-------compression level (1-9, 999)             !! ----Memory level (memory required for compression)

   In a compression type, different compression levels and memory levels of the data structure are the same, so the Lzo1x decompression method, can be applied to all lzo1x-** corresponding compression algorithm. LZO1 Classification: Compression data structure using strict Btye check LZO2 classification: Using bit displacement method, decompression speed is relatively slow.
6. Why are there so many algorithms?

   Because of a lot of historical versions, and I want to be able to provide good backwards compatibility, so have so many algorithms. Don't worry about the size of the whole library, using one of these algorithms will only add a few KiB to your program.

   If you just want to add a simple compression function to your program, you can view the Minilzo section in Minilzo/readme. LZO, you can view more slender information.

8. What kind of algorithm should I use?

   Lzo1x is a better choice in many cases. So, the pursuit of speed can use Lzo1x-1, generate pre-compressed files can use lzo1x-999, if you have a lot of memory, you can use Lzo1x-1 (11) or lzo1x-1 (12).

   Of course there are a lot of options here, you can choose according to your own needs. You can also try lzo1y or lzo1f and so on.

9. What is the difference between each of these methods when extracting?

   We still use lzo1x as an example to illustrate:

   • Lzo1x_decompress

     He is able to provide a faster decompression rate, which is a standard way to decompress. But he does not provide additional relevant data security checks, and the data he handles by default is the finished compressed data. If the data is damaged for a variety of reasons, such as hard drive damage or errors in the transmission, then he may have a serious error directly terminating your program.

   • Lzo1x_decompress_safe

     This is a safe way to decompress, but he will be slower. He captures the data in the decompression and returns the relevant error message, avoiding a serious error.

   • Lzo1x_decompress_asm

     A method similar to lzo1x_decompress, but is implemented through assembly.

   • Lzo1x_decompress_asm_safe

     A method similar to Lzo1x_decompress_safe, but is implemented through assembly.

   • Lzo1x_decompress_asm_fast

     Similar to the LZO1X_DECOMPRESS_ASM approach, but can provide faster speeds. To increase speed, he adds 3 BYTE of data at the end of the extracted data. This is because the data is processed as a unit in this way.

     When you extract data from one memory block to another, you need to provide more than 3 bytes of data storage space. If you want to extract a video content directly, do not use this method, because the last addition of 3 BYTE content will be treated as the display content.

   • Lzo1x_decompress_asm_fast_safe

     This is a secure version of Lzo2x_decompress_asm_fast.

   Here are some things to keep in mind:

   • When using a security type decompression method, the Dst_len parameter is used to describe the maximum amount of data that the DST parameter can be placed on.
   • If you want to confirm that your compressed data is safe, use a check such as MD5, which is not required if you are using a secure type. Because in a secure manner, any error is snapped back and returned.
   • The assembly version can only be used in the i386 schema, with specific information to view the Asm/i386/ooreadme. TXT file to view relevant content.
   • You might want to test if the assembly version is faster than the C language version, and some compilers can do a good job of optimizing it, making the C language version work well.
11. What is optimized for processing?

    Compression uses a heuristic approach, where they sometimes record information, which does not increase the compression ratio. The optimization places the unnecessary information, to improve the decompression speed.

    This also does not deal with content that may record changes in compressed content when decompressed. The compressed data is rearranged in some bits, which does not cause the capacity changes of the compressed data.

    After optimized processing, the decompression speed can be improved 1-3%. Optimization has no more effect to improve.

13. I need a faster decompression speed.

    In many RISC processing (for example MIPS), the delivery speed for 32-bit WORD is faster than BYTE, which can effectively improve the decompression speed. So after verifying that the rest of the work is working properly, you can use lzo_aligned_ok_4 to improve efficiency when lzo1x and lzo1y two algorithms are decompressed. Change the content in the config.h, and recompile everything to use it.

    In the i386 architecture, you can try to use the assembly version.

15. How do I reduce memory usage when compressing or decompressing?

    If you can intelligently control the range of data, then you can use compressed memory when extracting it. This will effectively remove the memory that holds the compressed data. This requires you to understand the basics of executable packages.

    Of course you can also partially reuse the buff when compressing

    In the example of EXAMPLES/OVERLAP.C, you can see more detailed content.

16. Can I get a guide on pre-compression?

    In the case of the algorithm lzo1x-999:

    Pre-Compression steps:

    1 Call lzo_init ()   2 Call lzo1x_999_compress ()   3 Call Lzo1x_optimze ()   4 make a 32-length MD5  for the compressed data 5 Save the compressed data and the corresponding MD5 in a file   6 If you are not at ease, you can now perform a decompression verification  

    Decompression steps:

    1 Call lzo_init ()   2 load your compressed data and the corresponding MD5   3 verify your compressed data  with MD5 4 Decompression  

    A more detailed example can be seen in the code for EXAMPLES/PRECOMP.C and examples/precomp2.c.
18. How much will my data grow after compression?

    LZO for non-compressible data, a bit of capacity is added. I haven't calculated the exact numbers, but here are the most wasteful formulas:

    Compression algorithm: lzo1,lzo1a,lzo1b,lzo1c,lzo1f,lzo1x,lzo1y,lzo1z

    1  -  - 3  

    This is approximately 106% of the original data.

    Compression algorithm: LZO2A

    1 8  - 3  

Original file:

• http://www.oberhumer.com/opensource/lzo/lzofaq.php

LZO Use and introduction