Fast decompressed lossless compression algorithm FRZ

Fast decompressed lossless compression algorithm FRZ

Author: HouSisong@GMail.com 3.03.06.16

FRZ (fast decompress arithmetic) is a lossless compression algorithm, which only aims to optimize the decoding speed!

Project source codeAddress: https://github.com/sisong/FRZ

 

To further compress the encoding data generated by the FRG image format, a lossless compression algorithm that can be quickly decompressed is required (you can focus only on the decompression speed rather than the compression time and memory space usage );

(FRG introduction:
Http://blog.csdn.net/housisong/article/details/9077059 );

You cannot select zip, lzma, or other common data compression algorithms because the extraction of these algorithms is too slow for FRG requirements (otherwise, you do not need to develop the FRG format );

To this end, a lossless compression algorithm FRZ that can be quickly decompressed has been developed. The compression ratio is moderate and the encoding format is simple (the decoder is also extremely simple ), decoding is extremely fast (it can even decode more than 1 GB of data per second !)

I tried the lzo algorithm specially designed for fast codec applications. Generally, the compression ratio and decompression speed curve are worse than FRZ;

Compression rate and decoding performance test:

Test File: http://www.maximumcompression.com/data/files/index.html
ZIP: used zlib library, version zlib_version "1.2.5"; parameter:, 9
Lzo: lzo-2.06, lzo1x_1 _ compress, lzo1x_11 _ compress,..., lzo1x_999_compress
FRZ: frz1_compress frz2_compress; parameter: 0, 1, 2, 4, 7
========================================================== ========================================================== ================
Filename filesize compressproc compressratio compressspeed uncompressspeed
Bytes --------------------------------------------------------------------------------------------
"World95.txt" 2.85013 M zlib_9 28.889% 13.3756 m/s 207.162 m/s
"World95.txt" 2.85013 M zlib_6 29.2304% 21.5702 m/s 206.421 m/s
"World95.txt" 2.85013 M zlib_1 37.5147% 54.8197 m/s 184.074 m/s

"World95.txt" 2.85013 M lzo1x_999 31.6574% 6.88294 m/s 488.259 m/s
"World95.txt" 2.85013 M lzo1x_1 51.5552% 314.079 m/s 438.298 m/s
"World95.txt" 2.85013 M lzo1x_15 52.7366% 323.153 m/s 436.311 m/s
"World95.txt" 2.85013 M lzo1x_12 55.4495% 332.623 m/s 433.35 m/s
"World95.txt" 2.85013 M lzo1x_11 60.864% 335.144 m/s 445.085 m/s

"World95.txt" 2.85013 M frz1_0 27.2492% 3.48432 m/s 839.173 m/s
"World95.txt" 2.85013 M frz1_1 27.7753% 4.77948 m/s 888.213 m/s
"World95.txt" 2.85013 M frz1_2 28.6499% 5.72374 m/s 939.491 m/s
"World95.txt" 2.85013 M frz1_4 31.5399% 6.40978 m/s 1059.31 m/s
"World95.txt" 2.85013 M frz3167 37.8838% 6.59552 m/s 1353.37 m/s

"World95.txt" 2.85013 M frz2_0 25.9137% 3.25779 m/s 642.307 m/s
"World95.txt" 2.85013 M frz2_1 26.6575% 4.52068 m/s 675.542 m/s
"World95.txt" 2.85013 M frz2_2 27.7533% 5.51231 m/s 717.292 m/s
"World95.txt" 2.85013 M frz2_4 31.1589% 6.23411 m/s 846.487 m/s
"World95.txt" 2.85013 M frz2_7 38.3927% 6.48505 m/s 1122.84 m/s
Bytes --------------------------------------------------------------------------------------------
"Ohs.doc" 3.9751 M zlib_9 24.0346% 20.8379 m/s 328.709 m/s
"Ohs.doc" 3.9751 M zlib_6 24.2109% 41.9468 m/s 325.815 m/s
"Ohs.doc" 3.9751 M zlib_1 26.1445% 78.2748 m/s 302.806 m/s

"Ohs.doc" 3.9751 M lzo1x_999 24.821% 14.8479 m/s 1668.89 m/s
"Ohs.doc" 3.9751 M lzo1x_1 30.4144% 1063.99 m/s 1756.48 m/s
"Ohs.doc" 3.9751 M lzo1x_15 30.5782% 1084.66 m/s 1771.64 m/s
"Ohs.doc" 3.9751 M lzo1x_12 30.9154% 1132.69 m/s 1783.67 m/s
"Ohs.doc" 3.9751 M lzo1x_11 31.4825% 1114.2 m/s 1799.86 m/s

"Ohs.doc" 3.9751 M frz1_0 22.5931% 7.54958 m/s 2380.45 m/s
"Ohs.doc" 3.9751 M frz1_1 22.7023% 7.9119 m/s 2503.14 m/s
"Ohs.doc" 3.9751 M frz1_2 22.896% 8.17576 m/s 2628.07 m/s
"Ohs.doc" 3.9751 M frz1_4 23.3212% 8.20975 m/s 2791.21 m/s
"Ohs.doc" 3.9751 M frz3167 24.3234% 7.76495 m/s 3170.95 m/s

"Ohs.doc" 3.9751 M frz2_0 22.4108% 7.38798 m/s 2027.87 m/s
"Ohs.doc" 3.9751 M frz2_1 22.5727% 7.68438 m/s 2135.81 m/s
"Ohs.doc" 3.9751 M frz2_2 22.8216% 7.97763 m/s 2248.16 m/s
"Ohs.doc" 3.9751 M frz2_4 23.3309% 8.04447 m/s 2439.09 m/s
"Ohs.doc" 3.9751 M frz2_7 24.4686% 7.60853 m/s 2866.19 m/s
Bytes --------------------------------------------------------------------------------------------
"FP. log" 19.662 M zlib_9 6.46612% 41.2731 m/s 444.306 m/s
"FP. log" 19.662 M zlib_6 7.03042% 74.2127 m/s 434.261 m/s
"FP. log" 19.662 M zlib_1 9.39676% 165.841 m/s 389.226 m/s

"FP. log" 19.662 M lzo1x_999 7.8425% 19.9185 m/s 1950.53 m/s
"FP. log" 19.662 M lzo1x_1 13.0122% 1183.41 m/s 1464.03 m/s
"FP. log" 19.662 M lzo1x_15 13.0558% 1202.05 m/s 1461.61 m/s
"FP. log" 19.662 M lzo1x_12 13.1413% 1232.84 m/s 1453.65 m/s
"FP. log" 19.662 M lzo1x_11 13.3616% 1207.43 m/s 1439.56 m/s

"FP. log" 19.662 M frz1_0 5.84649% 5.75424 m/s 2110.35 m/s
"FP. log" 19.662 M frz1_1 5.86878% 6.53614 m/s 2132.9 m/s
"FP. log" 19.662 M frz1_2 5.8881% 6.84821 m/s 2148.49 m/s
"FP. log" 19.662 M frz1_4 5.9365% 7.09288 m/s 2204.47 m/s
"FP. log" 19.662 M frz3167 5.99979% 7.46442 m/s 2203.12 m/s

"FP. log" 19.662 M frz2_0 5.62524% 5.84176 m/s 1915.42 m/s
"FP. log" 19.662 M frz2_1 5.65421% 6.41743 m/s 1932.08 m/s
"FP. log" 19.662 M frz2_2 5.67773% 7.05402 m/s 1944.85 m/s
"FP. log" 19.662 M frz2_4 5.73099% 7.26351 m/s 1982.19 m/s
"FP. log" 19.662 M frz2_7 5.80445% 7.38424 m/s 1987.81 m/s
Bytes --------------------------------------------------------------------------------------------
"A10.jpg" 0.80344 M zlib_9 99.9226% 25.8326 m/s 519.215 m/s
"A10.jpg" 0.80344 M zlib_6 99.9226% 25.8589 m/s 520.691 m/s
"A10.jpg" 0.80344 M zlib_1 99.89% 27.1758 m/s 486.591 m/s

"A10.jpg" 0.80344 M lzo1x_999 100.499% 9.17407 m/s 5813.29 m/s
"A10.jpg" 0.80344 M lzo1x_1 100.38% 1457.13 m/s 10242.1 m/s
"A10.jpg" 0.80344 M lzo1x_15 100.38% 1494.81 m/s 10230.9 m/s
"A10.jpg" 0.80344 M lzo1x_12 100.38% 1504.7 m/s 10071.4 m/s
"A10.jpg" 0.80344 M lzo1x_11 100.38% 1510.2 m/s 10070.8 m/s

"A10.jpg" 0.80344 M frz1_0 99.9875% 6.53883 m/s 14141 m/s
"A10.jpg" 0.80344 M frz1_1 99.9858% 6.47823 m/s 14127.4 m/s
"A10.jpg" 0.80344 M frz1_2 99.9858% 6.60292 m/s 14165.1 m/s
"A10.jpg" 0.80344 M frz1_4 99.986% 6.60659 m/s 14109.4 m/s
"A10.jpg" 0.80344 M frz3167 99.9892% 6.61174 m/s 14154.8 m/s

"A10.jpg" 0.80344 M frz2_0 100.126% 6.32229 m/s 13233.8 m/s
"A10.jpg" 0.80344 M frz2_1 100.132% 6.34454 m/s 13409.2 m/s
"A10.jpg" 0.80344 M frz2_2 100.133% 6.34467 m/s 13338.1 m/s
"A10.jpg" 0.80344 M frz2_4 100.134% 6.33967 m/s 13420.6 m/s
"A10.jpg" 0.80344 M frz2_7 100.137% 6.33445 m/s 13352.9 m/s
Bytes --------------------------------------------------------------------------------------------
"Rafale.bmp" 3.95719 M zlib_9 30.239% 5.14433 m/s 202.057 m/s
"Rafale.bmp" 3.95719 M zlib_6 30.4061% 11.3986 m/s 201.291 m/s
"Rafale.bmp" 3.95719 M zlib_1 37.5053% 50.0337 m/s 175.55 m/s

"Rafale.bmp" 3.95719 M lzo1x_999 36.2311% 2.66643 m/s 470.544 m/s
"Rafale.bmp" 3.95719 M lzo1x_1 55.4084% 323.246 m/s 427.893 m/s
"Rafale.bmp" 3.95719 M lzo1x_15 55.8511% 331.818 m/s 426.133 m/s
"Rafale.bmp" 3.95719 M lzo1x_12 56.7154% 334.466 m/s 422.875 m/s
"Rafale.bmp" 3.95719 M lzo1x_11 58.0779% 336.643 m/s 430.853 m/s

"Rafale.bmp" 3.95719 M frz1_0 36.385% 4.00348 m/s 627.713 m/s
"Rafale.bmp" 3.95719 M frz1_1 37.3569% 5.47687 m/s 664.003 m/s
"Rafale.bmp" 3.95719 M frz1_2 39.5831% 6.14429 m/s 724.264 m/s
"Rafale.bmp" 3.95719 M frz1_4 46.6967% 6.71949 m/s 938.249 m/s
"Rafale.bmp" 3.95719 M frz3167 57.6092% 7.0609 m/s 1436.36 m/s

"Rafale.bmp" 3.95719 M frz2_0 34.0961% 3.74757 m/s 508.362 m/s
"Rafale.bmp" 3.95719 M frz2_1 35.3388% 5.22443 m/s 538.239 m/s
"Rafale.bmp" 3.95719 M frz2_2 37.9872% 5.93088 m/s 589.594 m/s
"Rafale.bmp" 3.95719 M frz2_4 46.1538% 6.59884 m/s 744.604 m/s
"Rafale.bmp" 3.95719 M frz2_7 58.2903% 6.91874 m/s 1192.42 m/s
Bytes --------------------------------------------------------------------------------------------
"Flashmxbench" 4.31723 M zlib_9 84.5967% 20.8444 m/s 265.313 m/s
"Flashmxbench" 4.31723 M zlib_6 84.6587% 23.9056 m/s 264.983 m/s
"Flashmxbench" 4.31723 M zlib_1 86.036% 30.1846 m/s 263.099 m/s

"Flashmxbench" 4.31723 M lzo1x_999 85.0036% 8.56335 m/s 1562.94 m/s
"Flashmxbench" 4.31723 M lzo1x_1 91.7292% 1632.04 m/s 2366.72 m/s
"Flashmxbench" 4.31723 M lzo1x_15 91.7815% 1693.88 m/s 2361.23 m/s
"Flashmxbench" 4.31723 M lzo1x_12 91.8679% 1730.09 m/s 2367.06 m/s
"Flashmxbench" 4.31723 M lzo1x_11 92.1046% 1767.16 m/s 2382.23 m/s

"Flashmxbench" 4.31723 M frzda-0 84.5813% 3.23082 m/s 2297.54 m/s
"Flashmxbench" 4.31723 M frz1_1 84.774% 3.40124 m/s 2453.96 m/s
"Flashmxbench" 4.31723 M frz1_2 85.1791% 3.48444 m/s 2665.7 m/s
"Flashmxbench" 4.31723 M frz1_4 86.4924% 3.53592 m/s 3397.22 m/s
"Flashmxbench" 4.31723 M frz3167 87.9702% 3.54694 m/s 4248.99 m/s

"Flashmxbench" 4.31723 M frz2_0 84.1174% 3.17844 m/s 1894.33 m/s
"Flashmxbench" 4.31723 M frz2_1 84.3822% 3.34307 m/s 2021.97 m/s
"Flashmxbench" 4.31723 M frz2_2 84.9003% 3.43627 m/s 2190.38 m/s
"Flashmxbench" 4.31723 M frz2_4 86.4858% 3.48984 m/s 2797.32 m/s
"Flashmxbench" 4.31723 M frz2_7 88.0997% 3.51169 m/s 3649.39 m/s
Bytes --------------------------------------------------------------------------------------------
"Vcfiu. HLP" 3.93049 M zlib_9 20.3679% 7.16392 m/s 252.321 m/s
"Vcfiu. HLP" 3.93049 M zlib_6 20.5657% 22.146 m/s 249.353 m/s
"Vcfiu. HLP" 3.93049 M zlib_1 25.5187% 72.1689 m/s 224.203 m/s

"Vcfiu. HLP" 3.93049 M lzo1x_999 22.571% 5.01896 m/s 709.644 m/s
"Vcfiu. HLP" 3.93049 M lzo1x_1 34.1043% 448.054 m/s 591.329 m/s
"Vcfiu. HLP" 3.93049 M lzo1x_15 34.2533% 453.702 m/s 589.056 m/s
"Vcfiu. HLP" 3.93049 M lzo1x_12 34.4965% 459.623 m/s 589.285 m/s
"Vcfiu. HLP" 3.93049 M lzo1x_11 35.0444% 456.899 m/s 591.464 m/s

"Vcfiu. HLP" 3.93049 M frz1_0 23.6271% 4.83119 m/s 840.87 m/s
"Vcfiu. HLP" 3.93049 M frz1_1 24.0077% 5.94413 m/s 902.439 m/s
"Vcfiu. HLP" 3.93049 M frz1_2 24.891% 7.44206 m/s 988.71 m/s
"Vcfiu. HLP" 3.93049 M frz1_4 27.6316% 8.55562 m/s 1209.42 m/s
"Vcfiu. HLP" 3.93049 M frz3167 32.405% 8.95633 m/s 1605.68 m/s

"Vcfiu. HLP" 3.93049 M frz2_0 21.9619% 4.47159 m/s 685.83 m/s
"Vcfiu. HLP" 3.93049 M frz2_1 22.6251% 5.57507 m/s 732.31 m/s
"Vcfiu. HLP" 3.93049 M frz2_2 23.8061% 7.14303 m/s 798.351 m/s
"Vcfiu. HLP" 3.93049 M frz2_4 27.1751% 8.29691 m/s 973.524 m/s
"Vcfiu. HLP" 3.93049 M frz2_7 32.5823% 8.77274 m/s 1347.61 m/s
Bytes --------------------------------------------------------------------------------------------
"Acrord32.exe" 3.69147 M zlib_9 44.6456% 8.50206 m/s 163.41 m/s
"Acrord32.exe" 3.69147 M zlib_6 44.7563% 18.0917 m/s 162.234 m/s
"Acrord32.exe" 3.69147 M zlib_1 47.4641% 41.0574 m/s 151.194 m/s

"Acrord32.exe" 3.69147 M lzo1x_999 48.8463% 6.0816 m/s 410.524 m/s
"Acrord32.exe" 3.69147 M lzo1x_1 61.4381% 389.186 m/s 589.566 m/s
"Acrord32.exe" 3.69147 M lzo1x_15 61.8725% 402.743 m/s 605.72 m/s
"Acrord32.exe" 3.69147 M lzo1x_12 62.56% 422.427 m/s 634.055 m/s
"Acrord32.exe" 3.69147 M lzo1x_11 63.7072% 437.486 m/s 679.48 m/s

"Export rd32.exe" 3.69147 M frz1_0 56.1619% 3.46317 m/s 561.553 m/s
"Acrord32.exe" 3.69147 M frz1_1 57.8987% 4.7477 m/s 758.901 m/s
"Acrord32.exe" 3.69147 M frz1_2 60.339% 5.53802 m/s 981.424 m/s
"Acrord32.exe" 3.69147 M frz1_4 65.351% 5.89063 m/s 1439.6 m/s
"Export rd32.exe" 3.69147 M frz3167 70.7947% 5.98325 m/s 2187.27 m/s

"Export rd32.exe" 3.69147 M frz2_0 52.7803% 3.23662 m/s 460.935 m/s
"Export rd32.exe" 3.69147 M frz2_1 55.7622% 4.5434 m/s 601.709 m/s
"Acrord32.exe" 3.69147 M frz2_2 58.9634% 5.37978 m/s 763.151 m/s
"Acrord32.exe" 3.69147 M frz2_4 64.8701% 5.75913 m/s 1139.32 m/s
"Export rd32.exe" 3.69147 M frz2_7 70.8402% 5.86586 m/s 1839.38 m/s
Bytes --------------------------------------------------------------------------------------------
"Mso97.dll" 3.60719 M zlib_9 57.8448% 9.25576 m/s 140.143 m/s
"Mso97.dll" 3.60719 M zlib_6 57.8875% 15.5765 m/s 139.846 m/s
"Mso97.dll" 3.60719 M zlib_1 60.0198% 33.0382 m/s 128.829 m/s

"Mso97.dll" 3.60719 M lzo1x_999 61.9115% 6.00856 m/s 337.09 m/s
"Mso97.dll" 3.60719 M lzo1x_1 75.5749% 342.667 m/s 564.663 m/s
"Mso97.dll" 3.60719 M lzo1x_15 76.1348% 354.153 m/s 590.565 m/s
"Mso97.dll" 3.60719 M lzo1x_12 77.0051% 377.801 m/s 637.061 m/s
"Mso97.dll" 3.60719 M lzo1x_11 78.3231% 402.412 m/s 709.79 m/s

"Mso97.dll" 3.60719 M frz1_0 68.986% 2.57071 m/s 501.695 m/s
"Mso97.dll" 3.60719 M frz1_1 71.4181% 3.76127 m/s 746.503 m/s
"Mso97.dll" 3.60719 M frz1_2 74.6458% 4.38681 m/s 1053.22 m/s
"Mso97.dll" 3.60719 M frz1_4 80.2088% 4.78057 m/s 1754.09 m/s
"Mso97.dll" 3.60719 M frz3167 84.8929% 4.86758 m/s 3068.39 m/s

"Mso97.dll" 3.60719 M frz2_0 65.1535% 2.40022 m/s 415.354 m/s
"Mso97.dll" 3.60719 M frz2_1 69.2169% 3.60432 m/s 586.736 m/s
"Mso97.dll" 3.60719 M frz2_2 73.3673% 4.27331 m/s 815.739 m/s
"Mso97.dll" 3.60719 M frz2_4 79.8338% 4.68398 m/s 1389.87 m/s
"Mso97.dll" 3.60719 M frz2_7 84.8904% 4.71985 m/s 2568.53 m/s
Bytes --------------------------------------------------------------------------------------------
"English. DIC" 3.87901 M zlib_9 25.8048% 2.48666 m/s 200.218 m/s
"English. DIC" 3.87901 M zlib_6 25.8014% 14.6887 m/s 199.801 m/s
"English. DIC" 3.87901 M zlib_1 32.0235% 61.8219 m/s 188.032 m/s

"English. DIC" 3.87901 M lzo1x_999 35.8305% 2.46456 m/s 511.127 m/s
"English. DIC" 3.87901 M lzo1x_1 43.8161% 335.546 m/s 442.079 m/s
"English. DIC" 3.87901 M lzo1x_15 43.8511% 343.424 m/s 443.271 m/s
"English. DIC" 3.87901 M lzo1x_12 43.9007% 342.985 m/s 445.893 m/s
"English. DIC" 3.87901 M lzo1x_11 44.0295% 345.315 m/s 450.862 m/s

"English. DIC" 3.87901 M frz1_0 37.6059% 1.14149 m/s 491.934 m/s
"English. DIC" 3.87901 M frz1_1 38.2166% 2.30036 m/s 494.039 m/s
"English. DIC" 3.87901 M frz1_2 39.6943% 4.11223 m/s 495.177 m/s
"English. DIC" 3.87901 M frz1_4 47.8758% 6.69508 m/s 586.64 m/s
"English. DIC" 3.87901 M frz3167 74.0949% 6.91709 m/s 1335.55 m/s

"English. DIC" 3.87901 M frz2_0 33.401% 1.05071 m/s 396.83 m/s
"English. DIC" 3.87901 M frz2_1 34.0028% 2.17972 m/s 401.418 m/s
"English. DIC" 3.87901 M frz2_2 35.5348% 3.92784 m/s 406.464 m/s
"English. DIC" 3.87901 M frz2_4 44.8917% 6.4812 m/s 483.071 m/s
"English. DIC" 3.87901 M frz2_7 74.2199% 6.71249 m/s 1105.66 m/s
========================================================== ========================================================== ================

FRZ Compression Parameters support [0--16] to adjust the compression size and decoding speed. The default value is 4;
 

Frz1 encoding is divided into control data and uncompressed bytes;
Control Data by: "Type \ length [\ forward matching] \ Type \ length [\ forward matching] \… \ Type \ length [\ forward matching] "to encode;
"Type" occupies 1 bit, and 0 indicates the uncompressed data that is stored after it ("uncompressed byte data" stores multiple bytes continuously as is; there is no "forward matching" data in the control data). 1 indicates the compressed (alternative) data that is stored later (the distance data that matches forward is stored in the control data)
"Length" is a variable-length encoding. One Bit indicates whether the subsequent bytes are used for encoding, the remaining 7bit of the byte occupied by "type" is also used for Length Encoding (that is, if "length" value <= 6 bit, it can share a byte with "type );
"Forward matching" also uses variable-length encoding, which indicates copying "length" data from the current position to the forward distance. (This is the source of FRZ compression capability, the current data segment is replaced by the existing data segment );

As you can see, FRZ is an unusually simple data compression encoding format. Its decoder only requires about 40 lines of source code (the first version), but it can provide a good compression rate, and can quickly decompress data.
Simple code decoding also means possibleSpecial optimized versionIt is also easy to implement;
The current implementation of the encoding algorithm is also relatively simple. It is implemented using the oldest string algorithm of the suffix array; (time complexity O (N), spatial complexity 16 * n + O (1 ); space complexity should be optimized)
(Warning: Do not use FRZ for user-level runtime encoding or for real-time Encoding. FRZ encoding is slow and takes a huge amount of memory !)

PS: the reason for preventing FRZ from getting a greater compression ratio is that the storage of "length" and "forward matching" data may be further processed here, for example, to make statistics on the previous duplicate data and generate a shorter code according to the probability (or maintain a dynamic code table according to certain rules), you only need to store the encoding for the control data; however, such a design may have the risk of returning to the design idea of conventional compression algorithms, or may sacrifice the decoding performance. (but there is still an attractive implementation possibility !)