garbled processing of web crawler

Original address: http://www.cnblogs.com/agileblog/p/3615250.html

On the crawler garbled there are many groups of friends of all kinds of problems, below a simple summary of the web crawler garbled processing. Note that here is not only Chinese garbled, but also some such as Japanese, Korean, Russian, Tibetan and other garbled processing, because their solution is consistent, so this unified description.

Web crawler, there are two options, one is the choice of Nutch, Hetriex, and the second is self-writing crawler, the two in the processing of garbled, the principle is consistent, but the former processing garbled, to understand the source code after the modification can be, so to scrap a bit, and the latter is more free and convenient, can be processed in the encoding process. This is also a lot of people in the framework of the crawler will appear a variety of garbled, the reason why, like more mature nutch in the processing of garbled is also relatively simple, so there will be garbled, so need two times to develop to really solve garbled problem.

1, the cause of the network crawler garbled

Source page encoding and crawl down after the encoding conversion inconsistent. If the source Web page is GBK encoded byte stream, and we grasp to remove the program directly using UTF-8 encoding and output to the storage file, which will inevitably cause garbled, that is, when the source page encoding and crawl down after the program directly using processing encoding consistent, there will be no garbled, In this case, the uniform character encoding will not be garbled. Attention to distinguish the source network code A, the program directly uses the encoding B, the uniform conversion character encoding C.

A, is the server-side encoding of the Web page

B, the data fetched, the original case is a byte array, it is encoded by a, only b=a, can be guaranteed not garbled, otherwise when the character set is incompatible, always garbled, this step is often used for testing.

C, unified transcoding refers to get the original code of the Web page after a, then the unified code, mainly in order to unify the data of each page into a class of coding, often choose a larger utf-8 of the character set is advisable.

Each page has its own code, such as GBK, Utf-8, Iso8859-1, as well as Japanese JP system coding, Western Europe, Russian and other codes are different, when the crawl will always expand the various codes, and some crawler is the Web page for simple code recognition and then unified coding, Some do not make the source of the judgment of the Web page directly unified according to Utf-8 to deal with, this is obviously will cause garbled situation.

2, garbled solution method

It is very easy to find a solution based on the reason.

(1) Determine the source page encoding a

Encoding a tends to be in three locations in the Web page, the content of the HTTP header, the meta charset of the Web page, and the document definition in the page header. When you get the source page encoding, you can determine the three parts of the data in turn, and from there, the priority is also the same.

In theory this is true, but some of the domestic sites are very inconsistent with the norms, such as written GBK, is actually utf-8, some write utf-8, but the actual is GBK, of course, this is a small number of sites, but does exist. Therefore, in determining the page encoding, the special case should be handled, such as Chinese check, the default encoding and other strategies.

There is also a situation, is the above three are not encoded information, the general use of Cpdetector and other third-party web page Coding Intelligent Identification Tool to do, its principle is the statistical byte array of the characteristics of the probability to calculate the actual code, there is a certain degree of accuracy, but I actually found that the accuracy rate is very limited.

But after synthesizing the above three kinds of coding confirmation mode, almost can completely solve the Chinese garbled problem, in my network crawler system based on the nutch1.6 two times, the correct code can reach 99.99%, also proved the feasibility of the above method strategy.

(2) Program to restore the source Web page data by encoding B

Obviously, the B here is to be equal to a, in Java, such as the resulting byte array of the source page is Source_byte_array, then converted to string Str=new string (source_byte_array,b); That is, in memory the corresponding characters of these byte arrays are correctly encoded and can be displayed, at this time the printout is normal, this step is often used for debug or console output to do the test.

(3) Unified transcoding

The web crawler system has many sources of data, it is impossible to use the data, and then into its original data, if this is a waste of things. So the general crawler system to crawl down the results of a unified code, so that in the use of consistent external, easy to use. At this point, on the basis of (2), do a unified code conversion can be implemented in Java as follows

The byte array of the source Web page is Source_byte_array

Convert to a normal string: string Normal_source_str=new string (source_byte_array,c), which can be stored directly with the Java API, but is often not written directly to the string, Because the General crawler storage is multiple source pages stored in a file, so to record the byte offset, so the next step.

Then convert the resulting STR into a uniform encoded C-format byte array, then byte[] New_byte_array=normal_source_str.getbytes (c), you can write the array to the file using the Java IO API, and record the corresponding byte array offset, and so on, when really used, direct IO read.

garbled processing of web crawler