Making a simple crawler using node. js

Recently saw some knowledge of node. js, just like what to do with it. Because they like photography, often on the hummingbird net, so thinking about it, simply do a simple crawler to strip forum posts.

Go directly to the code.

1 varSYS = require ("sys"),2HTTP = require ("http"),3FS = require ("FS");4 varurl = "Http://newbbs.fengniao.com/forum/forum_125.html";5 vardata = "";6 //create regular, filter HTML7 varreg =/<div\s*class= "Recommendsub" >\s*<a\s*href= "(. *?)" \s*class= "pic" \s*title= "(. *?)" /G;8 varresult = [],9TimeStamp = 0;Ten //Create a request One varreq = http.request (URL,function(res) { A     //Setting the display encoding -TimeStamp =NewDate (). GetTime (); -Res.setencoding ("UTF8"); theRes.on (' Data ',function(chunk) { -Data + =Chunk; -     }); -     //after the response time, output data +Res.on (' End ',function () { -          while(match =reg.exec (data)) { + Result.push ({ ATitle:match[1], atUrl:match[2] -             }); -         } - WriteFile (result); -     }); - }); in //Send Request - req.end (); to functionWriteFile (r) { +     varstr = "; -      for(vari = 0, _len = r.length; i < _len; i++) { thestr + = R[i].title + ' \ n ' + r[i].url + ' \ n '; *     } $     //after the data stitching is finished, write the S.txt filePanax NotoginsengFs.writefile (' S.txt ', str,function(err) { -         if(err) { the             Throwerr; +         } AConsole.log (' time-consuming approx ' + math.ceil (NewDate (). GetTime ()-TimeStamp)/(+) + ' s '); theConsole.log (' data has been written to file '); +     }); -}

Here more attention is paid to the regular, we should pay attention to greed to match these points.

We named this JS file Spider.js, under the console run

Node Spider.js

After you can see the S.txt file generated, which is to get the post information.

That's it, I'm going to take the time to learn about node. js, and this guy is pretty funny.

The relevant JS I also uploaded to GitHub, interested students can clone to the local.

Address: Https://github.com/kbqncf/spider