Nodejs Crawl Web page encoding problem

Source: Internet
Author: User

Recently in the completion of the need to use the crawler system, page crawl this piece using the Nodejs language, in the beginning to write the page crawl program, for the GB2312 encoded page after the display is garbled, began to think that in Java reading a file when a code conversion work can be solved, But tried for a long while, the code will only become slower, and later guess, Nodejs request to the body of the page is transmitted by the binary stream information, if the wrong encoding to the binary information decoding, then the results of the pit is wrong, and then save this result to the file, the result is certainly still wrong, Therefore, through the above method, it is impossible to get the correct result.

Later the solution is, in the Nodejs, the binary data decoding, through the regular expression to obtain the correct encoding of the file, and then use the correct encoding to decode the binary data, and finally get the correct results.

In the request page information, some websites because the content of the page is too much, the content of the page is compressed, you can view the compression method used by headers[' Content-code '], for this kind of page, before parsing, must first the convection information to decompress, After the complete flow information is obtained, the information is decoded using the above method.

Nodejs in the code operation, using the Iconv-lite Library, compression and decompression using the Zlib library.

Nodejs Crawl Web page encoding problem

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.