Node. js crawls Chinese webpage garbled issues and solutions, node. js Chinese webpage

Node. js crawls Chinese webpage garbled issues and solutions, node. js Chinese webpage

When Node. js captures a non-UTF-8 Chinese webpage, garbled characters may occur. For example, if Netease's homepage code is gb2312, garbled characters may occur during crawling.
Copy codeThe Code is as follows:
Var request = require ('request ')
Var url = 'HTTP: // www.163.com'

Request (url, function (err, res, body ){
Console. log (body)
})

You can use iconv-lite to solve this problem.

Install
Copy codeThe Code is as follows:
Npm install iconv-lite

At the same time, we can modify the user-agent to prevent website blocking:
Copy codeThe Code is as follows:
Var originRequest = require ('request ')
Var iconv = require ('iconv-lite ')
Var headers = {
'User-agent': 'mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/8080'
}

Function request (url, callback ){
Var options = {
Url: url,
Encoding: null,
Headers: headers
}
OriginRequest (options, callback)
}

Request (url, function (err, res, body ){
Var html = iconv. decode (body, 'gb2312 ')
Console. log (html)
})

Garbled Problem Solving

Use cheerio to parse HTML

Cheerio can be simply and roughly understood as the jQuery selector on the server side. It is much more intuitive than regular expressions.

Install
Copy codeThe Code is as follows:
Npm install cheerio
Request (url, function (err, res, body ){
Var html = iconv. decode (body, 'gb2312 ')
Var $ = cheerio. load (html)
Console. log ($ ('h1 '). text ())
Lele.log((('h1'{.html ())
})

Output:
Copy codeThe Code is as follows:
Netease
Netease

As a matter of question, the code output by ('h1'{.html () is Unicode encoded, and Netease has become Netease, which brings us some trouble in character processing.

Solve the "garbled" problem of cheerio. html ()
You can disable this function to convert entity encoding.
Copy codeThe Code is as follows:
Var $ = cheerio. load (html)

Change
Copy codeThe Code is as follows:
Var $ = cheerio. load (html, {decodeEntities: false })

The complete code is as follows:
Copy codeThe Code is as follows:
Var originRequest = require ('request ')
Var cheerio = require ('cheerio ')
Var iconv = require ('iconv-lite ')
Var headers = {
'User-agent': 'mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/8080'
}

Function request (url, callback ){
Var options = {
Url: url,
Encoding: null,
Headers: headers
}
OriginRequest (options, callback)
}

Var url = 'HTTP: // www.163.com'

Request (url, function (err, res, body ){
Var html = iconv. decode (body, 'gb2312 ')
Var $ = cheerio. load (html, {decodeEntities: false })
Console. log ($ ('h1 '). text ())
Lele.log((('h1'{.html ())
})