Node. js crawls and analyzes whether the webpage content contains any special content. js file _ node. js

Nodejs obtains the webpage content and binds the data event. The obtained data is matched several times. If you want to globally match the content, wait until the request ends, perform operations on the accumulated global data in the end event. This article will introduce you to node. js captures and analyzes js files with or without special content on the webpage. If you need a friend, refer to nodejs to obtain the webpage content and bind the data event. The obtained data will be matched several times, if you want to globally match the content, wait until the request ends. In the end event, perform operations on the accumulated global data!

For example, if you want to find www.baidu.com on the page, simply put the code:

// Introduce the module var http = require ("http"), fs = require ('fs'), url = require ('url'); // write the file, write the results to different files var writeRes = function (p, r) {fs. appendFile (p, r, function (err) {if (err) console. log (err); else console. log (r) ;};}, // send a request, verify the content, and write the result to the file postHttp = function (arr, num) {console. log ('nth '+ num +! ") Var a = arr [num]. split ("-"); if (! A [0] |! A [1]) {return;} var address = url. parse (a [1]), options = {host: address. host, path: address. path, hostname: address. hostname, method: 'get', headers: {'user-agent': 'mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) chrome/38.0.2125.122 Safari/537.36 '} var req = http. request (options, function (res) {if (res. statusCode = 200) {res. setEncoding ('utf-8'); var data = ''; Res. on ('data', function (rd) {data + = rd;}); res. on ('end', function (q) {if (! ~ Data. indexOf ("www.baidu.com") {return writeRes ('. /no2.txt ', a [0] +' -- '+ a [1] +' \ n');} else {return writeRes ('. /has2.txt ', a [0] +' -- '+ a [1] + "\ n") ;}} else {writeRes ('. /error2.txt ', a [0] +' -- '+ a [1] +' -- '+ res. statusCode + '\ n') ;}}); req. on ('error', function (e) {writeRes ('. /error2.txt ', a [0] +' -- '+ a [1] +' -- '+ e +' \ n');}) req. end () ;}, // read the file and obtain the page to be crawled. openFile = function (pa Th, coding) {fs. readFile (path, coding, function (err, data) {var res = data. split ("\ n"); for (var I = 0, rl = res. length; I <rl; I ++) {if (! Res [I]) continue; postHttp (res, I) ;}})}; openFile ('./sites. log', 'utf-8 ');

You can understand the code above. If you are not clear about it, you are welcome to leave a message for me. It depends on your application in practice.

The following describes Nodejs's web page capturing capabilities.

First, PHP. Advantages: Web crawling and parsing of html frameworks can be used directly by various tools, which is easy to understand. Disadvantages: first, the speed and efficiency were very problematic. When I downloaded a movie poster, the crontab was executed on a regular basis without optimization. As there were too many php processes running, the memory went off. Then, the syntax is also very slow. There are too many keywords and symbols, which are not concise enough. This gives people a feeling that they have not been carefully designed, and it is very troublesome to write.

Node. js. The advantage is efficiency, efficiency, and efficiency. Because the network is asynchronous, it is basically as powerful as the concurrency of hundreds of processes, and the memory and CPU usage are very small, if you do not perform complex operations on the captured data, the bottleneck of the system is basically the bandwidth and I/O speed of the database such as MySQL. Of course, the opposite of the advantage is also a disadvantage. Asynchronous Network means that you need callback. In this case, if your business needs to be linear, for example, you must wait until the previous page is captured and get the data, in order to capture the next page, or even multi-layer dependencies, there will be a terrible multi-layer callback! At this time, the code structure and logic will be messy. Of course, you can use process control tools such as Step to solve these problems.

Finally, let's talk about Python. If you have no extreme requirements on efficiency, we recommend using Python! First, the Python syntax is very concise, and the same statement can be knocked many times on the keyboard. Then, Python is very suitable for data processing, such as packaging function parameters, list parsing, and matrix processing.