Node. js uses cheerio to create a simple web crawler example, node. jscheerio

Node. js uses cheerio to create a simple web crawler example, node. jscheerio

This article describes how to use cheerio to create a simple web crawler example for Node. js:

1. Objectives

1. Obtain the website title information
2. Output the obtained information to a new file.
3. Tool: cheerio. Use npm to download npm install cheerio
4. The use of cheerio APIS is basically the same as that of jQuery APIs.
5. If you are familiar with jQuery, cheerio will get started soon.

2. Code Section

Introduction: Get the sequence of the segment faultpage, and output the obtained sequence number to the pagetitle.txt file at the end.

Const https = require ('https'); const fs = require ('fs'); const cheerio = require ('cheerio '); const url = 'https: // segmentfault.com/'{https.get (url, (res) =>{ let html = ''; res. on ('data', (data) =>{ html + = data ;}); res. on ('end', () =>{ getPageTitle (html );});}). on ('error', () => {console. log ('webpage information error get');}); function getPageTitle (html) {const $ = cheerio. load (html); let chapters = $ ('. news _ item-title '); let data = []; let index = 0; let fileName = 'pageTitle.txt'; for (let I = 0; I <chapters. length; I ++) {let chapterTitle = $ (chapters [I]). find ('A '). text (). trim (); index ++; data. push ('\ n $ {index}, $ {chapterTitle}');} fs. writeFile (fileName, data, 'utf8', (err) =>{ if (err) {console. log ('fs file system failed to create new file ', err);} console. log ('successfully put the obtained title in the new file $ {fileName ')})}

The above is all the content of this article. I hope it will be helpful for your learning and support for helping customers.