Golang Learning Note 17 Crawler technology Roadmap, Python,java,nodejs,go language, scrapy mainstream framework Introduction

Golang Learning Note 17 Crawler technology Roadmap, Python,java,nodejs,go language, scrapy mainstream framework Introduction

Go Language Crawler framework:
Gocolly/colly,goquery,colly,chromedp,webloop,go_spider,pholcus
Https://github.com/hu17889/go_spider
Pholcus Wraith Spider heavyweight crawler software (with 3 operating interfaces)-Golang United States of China
https://www.golangtc.com/p/557473c9b09ecc2aa700000b
Henrylee2cn/pholcus_lib: Pholcus crawler rule Library for public maintenance
Https://github.com/henrylee2cn/pholcus_lib

Python, packaged frame scrapy, other commonly used, URLLIB2, parsing with the package of BeautifulSoup, with selenium. The above is a random climb.
Large-scale distributed crawl, the difficulty of one in the anti-crawling, dynamic IP pool, hit code, crawler behavior mode control, is a physical activity;
Another in the efficiency of crawling and landing, so to a certain scale distributed generally with Go/java/scala more

With Python scrapy, all platforms can run, Scrapy is the mainstream solution, all kinds of peripheral are very mature, crawling video python has ready-made packages

Python's Pyspider framework is relatively perfect, crawling a large number of sites, parsing a large number of pages distributed and background management is more convenient

Java crawler with Jsoup is also a good choice

Nodejs main framework is cheerio, crawler, Spiderman:

Use Nodejs to crawl the specified number of sites, with request plus cheerio is enough
Cheerio | Fast, flexible, and lean implementation of core jQuery designed specifically for the server.
https://cheerio.js.org/
crawler-npm
Https://www.npmjs.com/package/crawler
Ltebean/spiderman:a crawler with visualized Config board
Https://github.com/ltebean/spiderman