node. JS Web crawler

use node to develop a gadget to scan the state key laboratory of molecular reaction dynamics the information on the first three pages of the News list page (address as follows:Http://www.sklmr.dicp.ac.cn/list.php?tid =1) required to print out the news name, link address, release time.

For example, in the console printing

Hankley , "distinguished researcher of Chinese Academy of Sciences" program 2016-06-14 http://www.sklmr.dicp.ac.cn/show.php?id=521

new progress in the study of desorption kinetics of metal surface in our Chamber 2016-06-12 http://www.sklmr.dicp.ac.cn/show.php?id=520

Zhang Dayu lecture period : California Institute of Technology William A. Goddard III Professor 2016-05-04 http ://www.sklmr.dicp.ac.cn/show.php?id=519

/** * Design * First: Crawl one page to print one page. * Second: After all three pages have been crawled, stored in the array, unified printing. * The first method is more efficient. Use the first way. *//** * Ideas: * * Crawl Web site HTML content. Get the necessary content of the crawled HTML. Save the fetch to an array. Input the contents of the array to the console. */    varHTTP = require ('http');//introduces the Nodejs HTTP module, which is used to build HTTP services and use as httpclient.     varCheerio = require ('Cheerio');//It can be understood as a server-side jquery.  Use the same method as the client. //var promise = require (' promise '); //Process Control of asynchronous programming is more consistent with the programming habits of back-end programmers. //var url = 'http://www.sklmr.dicp.ac.cn/list.php?tid=1';//the URL to crawl, followed by stitching. //crawl information for each nodefunction Filterchapters (HTML) {var$ = cheerio.load (HTML);//Parse HTML content into DOM objects and filter the DOM like a jquery CSS selector query    varArticlelist = $ ('Td.text'). Find ('TR'); varArticlearr = []; Articlelist.each (function () {varCurele = $ ( This); vartitle = Curele.find ('A.title10'). Text (). replace (/\s*\r\n\s*/g,"");//Get article title    varTime = Curele.find ('Td.title11'). Text (). replace (/\s*\r\n\s*/g,"");//Get article Time    varhref ="http://www.sklmr.dicp.ac.cn/"+curele.find ('A.title10'). attr ('href');//Get article Links    if(title!=NULL&&title!="")//It 's a little difficult. Because DOM data and lines are of the same rank, and the line has only attributes that do not have an ID. Therefore, you must remove the TR null data in the line, otherwise you will print a portion of the empty data and error messages. Articlearr.push ({title:title, time:time, HR      Ef:href});}) returnArticlearr;}//printing information in the consolefunction Printcourseinfo (coursedata) {Coursedata.foreach (function (item) {varChaptertitle =Item.title; varChaptertime =Item.time; varChapterhref =Item.href; Console.log (Chaptertitle+"\ t"+chaptertime+"\ t"+chapterhref+"\ n"); });}//An arbitrary URL can be downloaded asynchronously (via the HTTP Get method), and when the download is complete, it invokes the callback function and passes the downloaded content as a parameter and outputs its contents to the console. function getpagelist (URL) {http.Get(URL, function (res) {varHTML ="'Res.on ('Data', function (data) {res.setencoding ('UTF8');//setting the buffer character setHTML + = data;//Splicing Buffer}) Res.on ('End', function () {//Handle the crawled content    varCoursedata=filterchapters (HTML);    Printcourseinfo (Coursedata); }). On ('Error', function (err) {Console.log ('error message:'+err)}) }//or request the first 3 pages of data. List = ['http://www.sklmr.dicp.ac.cn/list.php?tid=1','http://www.sklmr.dicp.ac.cn/list.php?tid=1&page=20','http://www.sklmr.dicp.ac.cn/list.php?tid=1&page=40']; for(varI=0;i<3; i++) {    varURL =List[i]; Getpagelist (URL);}/*or request the first 3 pages of data. Tid=1,tid=1&page=20,tid=1&page=40,tid=1&page=60 .... Such a pattern can be spliced URL.    var list = [];for (var i=0;i<=40;i+20) {var url = url+page=i; List.push (getpagelist (URL));} Call promise the following all method. A parameter is a collection of events. The promise will be executed asynchronously. But the final time to return is based on the most time-consuming request. Then (), you can accept two parameters (callback). The first parameter is a callback for success (resolved). The second parameter is a callback that performs the last operation failure (rejected).    Promise. All (AA). Then (function (data) {Console.dir (data); })*/

node. JS Web crawler