8 Most efficient Python crawler frameworks, how many have you used?

The small series collects some of the more efficient Python crawler frameworks. Share to everyone.

1.Scrapy

Scrapy is an application framework written to crawl Web site data and extract structural data. Can be applied in a series of programs including data mining, information processing, or storing historical data. With this framework, you can easily climb down data such as Amazon commodity information.

Project Address: https://scrapy.org/

2.PySpider

Pyspider is a powerful web crawler implemented in Python, it can be scripted on the browser interface, the scheduling of functions and real-time view of crawl results, the back end uses the common database for the storage of crawl results, but also can set tasks and task priority.

Project Address: Https://github.com/binux/pyspider

3.Crawley

Crawley can crawl the content of the corresponding website at high speed, support relational and non-relational database, data can be exported to JSON, XML and so on.

Project Address: http://project.crawley-cloud.com/

4.Portia

Portia is an open source visual crawler tool that allows you to crawl sites without any programming knowledge! Simply comment on the page you are interested in, Portia will create a spider to extract the data from a similar page.

Project Address: Https://github.com/scrapinghub/portia

5.Newspaper

Newspaper can be used to extract news, articles, and content analysis. Use multi-threading, support for more than 10 languages.

Project Address: Https://github.com/codelucas/newspaper

6.Beautiful Soup

Beautiful Soup is a python library that extracts data from HTML or XML files. It is able to use your favorite converter to achieve idiomatic document navigation, find, modify the way the document. Beautiful Soup will save you hours or even days of working hours.

Project Address: https://www.crummy.com/software/BeautifulSoup/bs4/doc/

7.Grab

Grab is a Python framework for building web scrapers. With grab, you can build a variety of complex web crawlers, from simple 5-line scripts to complex asynchronous web crawler tools that handle millions of of pages. Grab provides an API for performing network requests and processing received content, such as interacting with the DOM tree of an HTML document.

Project address: http://docs.grablib.org/en/latest/#grab-spider-user-manual

8.Cola

Cola is a distributed crawler framework that, for users, only needs to write a few specific functions without needing to focus on the details of the distributed operation. Tasks are automatically assigned to multiple machines, and the entire process is transparent to the user.

Project Address: Https://github.com/chineking/cola

Like the words concerned about the collection of comments forwarded more than the heart of the DA! Python Learning Exchange Enterprise-Goose-q-un-719-139-688 has a large number of project development and novice Instructional videos PDF Books of thousands of people waiting for you to join

8 Most efficient Python crawler frameworks, how many have you used?