A probe into node. JS Crawlers

Objective

In the Learning Network video and Cnode Novice access to the crawler, said is the crawler, in fact, did not use crawler-related third-party class library, mainly using the node. JS Basic module HTTP, Web analysis tools Cherrio. Use HTTP to get the URL path directly corresponding to the Web resource, and then use Cherrio analysis. Here I mainly put the network teaching video to provide the case of their own knocked on one side, deepen understanding. In the process of coding, the first time I took the JQ object directly with the foreach traversal, direct error, because JQ does not correspond to this method, only the JS array can be called.

Knowledge points

①:superagent grab the Web tool. I haven't used it for the time being.

②:cherrio Web Analytics tool, you can understand it as a service-side jquery, because the syntax is the same.

1. Crawl the entire page

2, the analysis of the data, I here is the MU-class network provides examples of examples for the implementation of the case.

Research on source code analysis of Reptiles

var http=require (' http '); var cheerio=require (' Cheerio '); var url= ' http://www.imooc.com/learn/348 ';/*************** Printed data structure [{chaptertitle: ', Videos:[{title: ', ID: '}]}]********************************/function Printcourseinfo (coursedata) {Coursedata.foreach (function (item) {var chaptertitle=item.chaptertitle;console.log ( Chaptertitle+ ' \ n '), Item.videos.forEach (function (video) {Console.log (' "' +video.id+ '" ' +video.title+ ' \ n ');})});} /************* analyze the data crawled from the Web page **************/function filterchapter (HTML) {var coursedata=[];var $=cheerio.load (HTML); var chapters=$ ('. Chapter '); Chapters.each (function (item) {var chapter=$ (this); var chaptertitle=chapter.find (' Strong '). text (); Find Chapter title var videos=chapter.find ('. Video '). Children (' Li '); var chapterdata={chaptertitle:chaptertitle,videos:[]}; Videos.each (function (item) {var video=$ (this). Find ('. Studyvideo '); var title=video.text (); var id=video.attr (' href ') . Split ('/video ') [1];chapterdata.videos.push ({title:title,id:id})}) Coursedata.push (Chapterdata);});    return coursedata;} Http.get (Url,function (res) {var html= '; Res.on (' Data ', function (data) {html+=data;}) Res.on (' End ', function () {var coursedata=filterchapter (HTML);p rintcourseinfo (coursedata);})}). On (' Error ', function () {Console.log (' Error getting course data ');})

Resources

Https://github.com/alsotang/node-lessons/tree/master/lesson3

http://www.imooc.com/video/7965

A probe into node. JS Crawlers