Come on, djvu!.

Ma Jian
Email:[email protected]
Published: 2010.05.21

Directory
First, DjVu technology
Second, the person who grasps the DjVu technology
Third, the people who play DjVu
Iv. Summary
PostScript: Me and DjVu

I would like to use this article to commemorate the 4 anniversary of dealing with DjVu!
======================= Simple dividing line ======================

After the public release of Djvutoy, there was a professional scanning outsourcing friends asked me about the future of DjVu, my answer is: if the status of DjVu is not changed, in commercial applications there is no future, can only be reduced to scan e-book lovers Hands of the plaything.

This so-called "status quo", refers to: a good technology, fell into a bunch of bad people.

This article is actually a collation of that conversation, purely opinion.

First, DjVu technology

I said DjVu technology is "pretty good" because:
1, DjVu based on the ISO/IEC 16485 MRC (Mixed Raster Content) model, in image layering, image coding, there are some unique, thus creating a DjVu "high compression rate" reputation, the reason I in the "DjVu to PDF" chapter II "theory" Section has been described in detail, here withheld.
2, because specially locates in the scanning document, does not consider the text display, therefore in the DjVu standard specification may omit the font and so on content, directly uses the UTF-8 code to represent the retrievable hidden text content, is more concise than the PDF and so on.

But DjVu technology is not perfect either:

1. Error rate of scanning resolution and lossy compression
DjVu usually uses JB2 compression when compressing text parts, which can be lossless or lossy. In the case of lossy, if the scanning resolution is not enough, the resulting djvu may be misspelled because of the bit error, so in the DjVu related technical documents I have seen, the scanning resolution should not be less than the DPI. The reason for the error, see "DjVu to PDF" article.
From the present situation, the error rate of Chinese characters is higher than that of alphabetic characters.

2. Lack of support for grayscale images
In the case of insufficient resolution (less than five DPI), if forced to simplify to black and white two colors, may appear the text is incomplete, and the use of grayscale images (such as Level 16 grayscale), you can balance the file length and display results, which is the practice of the PDG large chart. Unfortunately, the DjVu format itself is lacking in support for grayscale images:
A) if the conversion into a single-layer DjVu, the use of JB2 compression will obviously appear above the stroke of the problem, the use of IW44 compression in the large compression ratio will be blurred.
b) conversion to double DjVu obviously not, because the double djvu requires each shape can have only one color, and grayscale image each word has multilevel grayscale.
c) converted to three-layer DjVu, gray information is provided by the underlying image, in the use of large compression ratio (large scale reduction, low bit rate), the text will appear blurred, or even become a group of gray ash.
The solution is not no, but a little trouble, how the effect depends on technology and luck: zoom in (such as from the Zoom to the DPI), after processing subtractive black and white color, converted to JB2 compression of the single-layer DjVu.

3. Limitations of support for MRC models
DjVu based on the ISO/IEC 16485 MRC three layer model, but does not take into account the model's stripes, n-layer model, and so on, so the illustration if the proportion of the entire page is not small, after a large proportion of compression and then enlarge the display, the illustrations will always look blurry, affecting the perception.

4, batch processing difficult to get rid of human intervention
The compression rate of DjVu files, the visual effects after generation, etc., are in fact closely related to the accurate recognition of image types. The "accurate recognition of image type" is referred to here, that is, after encountering a scan of a good static image, it is accurate to determine whether it should be converted into single, double or three-layer DjVu, and determine the compression ratio of each layer. This kind of judgment is a kind of experience value to a large extent, so in commercial djvu production software, there are many kinds of preset values, need to be selected manually to obtain the best results. In other words, if all the documents are indiscriminately converted to three layers (document), it is unavoidable to encounter the gray image in front of them. All of this presents difficulties for fully automated batch DjVu generation. It is precisely this ability that business applications value.

5, congenital, internal and external different format standard
In my opinion, DjVu should not be regarded as a kind of "document format", but should be regarded as an "image format", because compared with the document format such as PDF, there are too many missing things, including multi-format text, vector graphics, multimedia, scripts, interactive forms, permissions control, security identification and so on, So it is only appropriate to represent scanned images, not for regular documents. However, even then, the DjVu format itself is not a fully publicized format (open format), such as the latest "Security DjVu" (SDJVU) format, only the business owner of the DjVu format Celertem the company, and is only supported by the products provided by the company and its affiliates, Other details unknown outside, so was ridiculed as "published format", the original address:
http://www.djvu.org/forum/phpbb/viewtopic.php?t=422
The author Jrile is the original Planetdjvu community founder and webmaster, in the DjVu community is also a famous.

6. Lack of tools and support
In this era, unless there is such a strong MS office, the application of a file format can not be separated from the strong support of the open source community. But the current DjVu is basically not an open format, in addition to the previous document format standard confusion, the latest, best DjVu format generation tools, SDKs are in the hands of Celertem company and its relations companies, these are money, and is not cheap.
At present, DjVu related to the free, open source of east, eventually can be traced back to Djvulibre (http://sourceforge.net/projects/djvu/), theoretically this project by the DjVu business owners (at the earliest, and then LizardTech) maintenance, but in fact, compared to a truly commercialized SDK, it's far worse:
A) Djvulibre does not have the most core function-image layering function, basically can only make single-layer DjVu, if you need to make three layer djvu, you need to use other code or tools to solve the problem of image layering. This is the only way to ensure that the business SDK is not replaced by free Djvulibre.
b) for JB2 compression, Djvulibre does not have a shared dictionary generation capability, and the resulting DjVu file may be larger than the DjVu file generated by the Commercial SDK.
c) The document I saw previously said that the source code for Djvulibre is from the LizardTech codebase, the same as the code used by the company's commercial SDK, but actually found the LizardTech release of the commercial SDK compiled, Only to know that this is not the case: the SDK documentation will tell you that this djvulibre is not djvulibre; The compiler will tell you that the Djvulibre source code you get is a lot less than the Djvulibre source code used by the Business SDK.
D) I do not know whether it is intentional or unintentional, although Djvulibre claims to adopt an intelligent memory management model, but the actual use of the people know that the source code of memory loopholes are countless (I myself spent 4 years to fill the hole), so I do not believe that any common sense of business software companies based on this set of source code To develop their own business applications.
e) The Djvulibre source code is based on the GPL, which has partly blocked its commercial application.
f) The two-Djvulibre owners (LizardTech, Celertem) have little support for the community, as I said earlier about the memory vulnerability, which was raised a few years ago but has not been admit. The support for other DjVu-related communities is even less, as Jrile, after closing Planetdjvu community, chooses some of the posts in the community and publishes the title "The Future of Djvu-revisited" in the djvu.org community:
http://www.djvu.org/forum/phpbb/viewtopic.php?t=526
In this post, his views on DjVu are enough to scare the average person.

In addition to the above flaws, technically speaking, the DjVu format itself is not irreplaceable: DjVu depends on the ISO/IEC 16485 MRC Three layer model, can also be applied to the PDF, and achieve similar compression ratios and visual effects, see my written "DjVu to PDF" article.

Interestingly, Djvulibre started with v3.5.21 and added DjVu to PDF in the Ddjvu example, but Libtiff source code was used. In other words, the DjVu is converted to TIFF first, then converted to PDF, supporting CCITT G4, JEPG, zip compression. Such conversion, not only completely regardless of the MRC three layer model, but also lost the original JB2, IW44 compression of high efficiency, the converted PDF will naturally add a lot of fat, and then some people can plausibly say: you see, DjVu converted into a PDF will become big bar!

So, DjVu technology, although good, but it is only "good".

Second, the person who grasps the DjVu technology

The DjVu format was presented at T/T in 1996, so the first 4 bytes of the file were "T".

At the DjVu, there are not many inputs, as can be seen from the source code and the SDK released at/T. In 2000, the DjVu format was sold to LizardTech, a company focused on document scanning.

LizardTech to DjVu file format specification, Djvulibre contribution to the obvious, unfortunately, a few years later also can not continue to, 2007 by the Japanese Celertem acquired. But strangely, on the Celertem company product homepage (http://www.celartem.com/en/products/), download homepage (http://www.celartem.com/en/download/), The version of Document Express seems a bit old, and the latest edition seems to be in another company Caminova's product page (http://www.caminova.jp/en/products/?src=products2.aspx), Download page (http://www.caminova.jp/en/downloads/), so I suspect that the real master of DjVu technology now is the new company Caminova,celartem is just playing capital operation.

The latest version of the DjVu SDK is also issued by Caminova, but is not found on the company's external HTTP website (http://www.caminova.jp/en/) and can only be accessed via HTTPS jumps over:
https://www.caminova.net/en/downloads/

It's really a bunch of companies that don't know what they're called.

Third, the people who play DjVu

For all these reasons, DjVu has been out of business for a long time, but has been sought after in some underground scanning e-book industries.

The reason for this is simple: underground scanning of e-books need to be shared over the Internet, so the file size is very sensitive, and for personal interest to scan the ebook, more than the money to complete the scan work more dedicated, very little less than the DPI, and even encourage the use of software amplification to the DPI (see the Scan and Share Tutorial "). In such a high resolution, the letter with DjVu can obtain a very high compression rate, lossy JB2 compression error probability is also very small.

Personal scanning of e-books to share, of course, is unlikely to buy expensive commercial DjVu software, and will not spend too much effort on DjVu technology, so I say "play" DjVu.

Russia's cracker world-famous, as far as I know "play" DjVu play the most brilliant is also the Russian e-book industry. As far as I know, the Russian cracker is not interested in the DjVu SDK, think this thing is always slower than commercial software, so much more keen to use commercial djvu production software, if need to customize, but also would rather from the latest version of business software to extract their own parts, The simplest example is the DjVu Small.

Under the influence of foreign electronic scanning e-book industry, there are also some people in China playing DjVu. But from my exposure to the situation, the domestic scanning e-book very few can reach the DPI, printing, scanning effect is worse than foreign, in this case, the impact of the use of lossy JB2 visual visibility. As for the 16-level grayscale PNG format of the large chart PDG directly converted to DjVu, but also the classic performance of technical idiots, I even the mind of jokes are too lazy.

Iv. Summary

In short, in the current situation, it is not a very responsible behavior to recommend to commercial customers to save scanned documents in the DjVu format, so I finally advised the man who was working on the scanning outsourcing to wait and see.

As for just want to "play", then play, pay attention to save the original scan format on the line, the other JB2 as far as possible to choose lossless compression, illustrations or color page compression ratio is not too harsh.

Lenin had a famous assertion that, for the purpose of greed, capitalists would even buy us the noose used to hang them. If the DjVu one-sided pursuit of compression ratio, sooner or later will be to their own neck of the pro-glove a noose.

PostScript: Me and DjVu

My dealings with DjVu began in 2006 with the PDG, since it was popular with the fast version PDG, while the decrypted rapid version of the PDG = PDG file header + DjVu file body.

But interestingly, in the rapid version of the PDG, the JB2 compressed black and white parts did not change, the use of IW44 compression (the core is the wavelet compression algorithm) color image is upside down, color interchange (red, blue interchange), so encountered the color of the rapid version of the PDG, Extract DjVu data directly from the decrypted file body, with the DjVu browser looks very awkward, can only be reversed back and then re-compressed again.

The reason for this is unclear, I guess with a company claiming to have "mastered the wavelet compression technology" related, do not do some hands and feet how to prove that they "master"? Of course, it may just be the software developers themselves confused, not clear the difference between RGB and BGR. In addition, Ssreader software developers do not seem to optimize the Djvulibre compilation options, resulting in DjVu decoding efficiency is a bit low, but the rapid version of the pixel size is not large, so it does not feel obvious.

Although in my first contact with the PDG, the rapid version of the PDG prevailed for a while, even some people put forward "Ning to express version, do not clear version", but with a number of genuine interest in the technology of the efforts of people, the rapid version of the PDG gradually revealed the truth, In particular, because of the low resolution of the use of lossy JB2 compression led to the emergence of the evidence of typos, at least in the Readfree, no one went to the public pursuit of the rapid version.

Shortly after contacting the PDG, I came into contact with another important source of DjVu e-book,--cadal (also known as the "US-China million"). From the cadal I did find some books that the PDG did not have, but cadal to browse the main online, offline browsing is not convenient. To this end, I began to develop Djvutoy software, the first few features of this software is actually for the download from the Cadal Book Service. But in Cadal began to add watermarks to the page, and use a large proportion of IW44 compressed text page, I finally unbearable, from now on Cadal no slightest interest.

Offline Browsing DjVu, I was destined not to use IE plugin, so chose Windjview. But in using Windjview browse some books, feel shining white a background is also very uncomfortable, so teeth in Unicornviewer added to DjVu support.

I sometimes help friends and colleagues find some information, but there are always some people do not want to use the unfamiliar browser, do not want me to convert DjVu into PDF. With my understanding of the principles of DjVu, I feel that the use of conventional printing methods, or the first turn into a common image format and then into a PDF, for the DjVu MRC model, JB2 compression, IW44 compression is a blasphemy, at least a irresponsible behavior. To this end, I spent half a year of spare time, research DjVu to PDF method, the core idea is to try to lossless, and keep the data flow length is basically consistent. The final result is the DjVu to PDF and the "to PDF" feature in Djvutoy.

This process let me to DjVu, the PDF format has its own understanding, never believe that the domestic some parrot nonsense, determined to say goodbye to DjVu, to the resolution of the PDF. But one of the thorns that always bothers me: I never mastered the image layering technology, so the universal image format goes directly to PDF, which always does not reach the compression rate of the DjVu.

This time, some selfless help appeared, finally have the Djvutoy in the "DjVu production" function. The feature may still be different from the latest version of the DjVu commercial software, but it's not as good as the previous version of commercial production software.

At this point, my fate with DjVu finally complete, so the title of this article is called "Don't, djvu! ”。

Finish

Come on, djvu!.