List of Python crawler tools, python Crawler

List of Python crawler tools, python Crawler
Network

• General

  • Urllib-network library (stdlib ).

  • Requests-network library.

  • Grab-network library (based on pycurl ).

  • Pycurl-network library (bound to libcurl ).

  • Urllib3-Python HTTP library, secure connection pool, support for file post, high availability.

  • Httplib2-network library.

  • RoboBrowser-a simple and Python-style Python library that allows you to browse Web pages without an independent browser.

  • Machicalsoup-a Python library that automatically interacts with the website.

  • Machize-stateful and programmable Web browser library.

  • Socket-underlying network interface (stdlib ).

  • Unirest for Python-Unirest is a lightweight HTTP library that can be used in multiple languages.

  • The HTTP/2 client of hyper-Python.

  • PySocks-SocksiPy is updated and actively maintained, including bug fixes and other features. Directly replace the socket module.

• Asynchronous

  • Treq-similar to requests APIs (based on twisted ).

  • Aiohttp-asyncio's HTTP client/server (PEP-3156 ).

Web Crawler framework

• Full-featured Crawlers

  • Grab-web crawler framework (based on pycurl/multicur ).

  • Scrapy-web crawler framework (based on twisted), does not support Python3.

  • Pyspider-a powerful crawler system.

  • Cola-a distributed crawler framework.

• Others

  • Portia-Scrapy-Based Visual crawler.

  • Restkit-HTTP resource toolkit for Python. It allows you to easily access HTTP resources and create objects around it.

  • Demiurge-PyQuery-based crawler microframework.

HTML/XML Parser

• General

  • Lxml-efficient HTML/XML processing library written in C language. Supports XPath.

  • Cssselect-parse the DOM tree and CSS selector.

  • Pyquery-parse the DOM tree and jQuery selector.

  • BeautifulSoup-inefficient HTML/XML processing library, pure Python implementation.

  • Html5lib-generate the DOM of the HTML/XML document according to the WHATWG specification. This specification is used in all browsers.

  • Feedparser-Parse RSS/ATOM feeds.

  • MarkupSafe-Provides secure escape strings for XML, HTML, and XHTML.

  • Xmltodict-a Python module that makes you feel like processing JSON when processing XML.

  • Xhtml2pdf-convert HTML/CSS to PDF.

  • Untangle-it is easy to convert an XML file into a Python object.

• Clear

  • Bleach-clear HTML (html5lib is required ).

  • Sanitize-brings clarity to the chaotic data world.

Text Processing

A library used to parse and operate simple text.

• General

• Difflib-(Python standard library) helps to compare differences.

• Levenshtein-quickly calculates Levenshtein distance and string similarity.

• Fuzzywuzzy-fuzzy string matching.

• Esmre-Regular Expression accelerator.

• Ftfy-automatically organizes Unicode text to reduce fragmentation.

• Conversion

• Unidecode-convert Unicode text to ASCII.

• Character encoding

• Uniout-print readable characters instead of escaped strings.

• Chardet-compatible with Python's 2/3 character encoder.

• Xpinyin-a database for converting Chinese characters into pinyin.

• Pangu. py-the spacing between CJK and alphanumeric characters in formatted text.

• Slug

• Awesome-slugify-a Python slugify library that can retain unicode.

• Python-slugify-a Python slugify library that can convert Unicode to ASCII.

• Unicode-slugify-a tool that can generate Unicode slugs.

• Pytils-a simple tool for processing Russian strings (including pytils. Transcoder. slugify ).

• General parser

• Python Implementation of PLY-lex and yacc parsing tools.

• Pyparsing-generate a syntax analyzer for a general framework.

• Person's name

• Python-nameparser-component of the parser name.

• Phone number

• Phonenumbers-parse, format, store and verify the international phone number.

• User proxy string

• Python-user-agents-the parser of the browser user proxy.

• HTTP Agent Parser-Python HTTP Proxy analyzer.

Processing specific format files

Parses and processes libraries of specific text formats.

• General

• Tablib-a module that exports data in the XLS, CSV, JSON, YAML, and other formats.

• Textract-extract text from various files, such as Word, PowerPoint, and PDF.

• Messytables-A Tool for parsing messy table data.

• Rows-a common data interface that supports many formats (CSV, HTML, XLS, and TXT are currently supported-more will be provided in the future !).

• Office

• Python-docx-read, query, and modify the Microsoft Word2007/2008 docx file.

• Xlwt/xlrd-read written data and format information from an Excel file.

• XlsxWriter-a Python module that creates the excel.xlsx file.

• Xlwings-a BSD-licensed library that can easily call Python in Excel, and vice versa.

• Openpyxl-a library for reading and writing Excel2010 XLSX/XLSM/xltx/XLTM files.

• Marmir-extract the Python data structure and convert it to a workbook.

• PDF

• PDFMiner-A Tool for extracting information from PDF documents.

• PyPDF2-a library that can split, merge, and convert PDF pages.

• ReportLab-allows quick creation of rich PDF documents.

• Pdftables-extract a table directly from a PDF file.

• Markdown

• Python-Markdown-a Markdown of John Gruber implemented in Python.

• Mistune-the fastest and fully functional Markdown pure Python parser.

• Markdown2-a fast Markdown that is fully implemented using Python.

• YAML

• PyYAML-a Python YAML parser.

• CSS

• Cssutils-a Python CSS library.

• ATOM/RSS

• Feedparser-a common feed parser.

• SQL

• Sqlparse-an unverified SQL statement analyzer.

• HTTP

• HTTP

• Http Request/Response Message parser implemented by HTTP-parser-C.

• Microformats

• Opengraph-a Python module used to parse Open Graph protocol labels.

• Portable execution body

• Pefile-a multi-platform module used to parse and process portable executable (PE) files.

• PSD

• Psd-tools-read the Adobe Photoshop PSD (PE) file to the Python data structure.

Natural Language Processing

Database that handles human language problems.

• NLTK-the best platform for compiling Python programs to process human language data.

• Pattern-Python network mining module. He has natural language processing tools, machine learning, and others.

• TextBlob-provides consistent APIs for deep NLP tasks. It is developed on the shoulders of giants Based on NLTK and Pattern.

• Jieba-Chinese word segmentation tool.

• SnowNLP-Chinese Text Processing library.

• Loso-another Chinese dictionary.

• Genius-Chinese Word Segmentation Based on Conditional Random Fields.

• Langid. py-an independent language recognition system.

• Korean-a Korean format library.

• Pymorphy2-Russian morphology analyzer (word-of-speech tagging + word-form change engine ).

• PyPLN-a distributed natural language processing channel written in Python. The goal of this project is to create a simple method to use NLTK to manage large language libraries through network interfaces.

Browser automation and Simulation

• Selenium-automated real browsers (Chrome, Firefox, opera, and IE ).

• Ghost. py-encapsulation of PyQt webkit (PyQT is required ).

• Spynner-encapsulation of PyQt webkit (PyQT is required ).

• Splinter-General API browser simulator (selenium web driver, Django client, Zope ).

Multiple Processing

• Threading-the thread running of the Python standard library. It is very effective for I/O-intensive tasks. It is useless for CPU-bound tasks because python GIL.

• Multiprocessing-the standard Python library runs multiple processes.

• Celery-asynchronous task queue/Job Queue Based on Distributed message transmission.

• The concurrent-futures-concurrent-futures module provides a high-level interface for calling asynchronous execution.

Asynchronous

Asynchronous Network programming library

• Asyncio-(Python standard library later than Python 3.4) asynchronous I/O, time loop, collaborative programs and tasks.

• Twisted-event-driven network engine framework.

• Tornado-a network framework and an Asynchronous Network Library.

• Pulsar-Python event-driven concurrency framework.

• Diesel-Python Green Event-based I/O framework.

• Gevent-A coroutine-based Python network library using greenlet.

• Eventlet-asynchronous framework supported by WSGI.

• Tomorrow-the wonderful modifier Syntax of asynchronous code.

Queue

• Celery-asynchronous task queue/Job Queue Based on Distributed message transmission.

• Huey-small multi-threaded task queue.

• Mrq-Mr. Queue-use the Python Distributed Job Queue of redis & Gevent.

• RQ-Redis-based lightweight task queue manager.

• Simpleq-a simple, infinitely scalable Queue Based on Amazon SQS.

• Python-gearman-Gearman Python API.

Cloud computing

• Picloud-run Python code on the cloud.

• Dominoup.com-cloud executes R, Python, and matlab code.

Email

Email resolution Library

• Flanker-email address and Mime parsing library.

• The Talon-Mailgun library is used to extract the quote and signature of a message.

Website and network address operations

Parse/modify the URL and network address library.

• URL

  • Furl-a small Python library that simplifies URL manipulation.

  • Purl-a simple unchangeable URL and a clean API for debugging and operations.

  • Urllib. parse-used to break the gap between components (addressing scheme, network location, path, etc.) in a uniform resource locator (URL) string. To combine components into a URL string, convert "relative URL" into an absolute URL, which is called "Basic URL ".

  • Tldextract-accurately detaches TLD from the URL registration domain and subdomain, and uses the public Suffix List.

• Network Address

  • Netaddr-Python library for displaying and manipulating network addresses.

Webpage content extraction

Library for extracting Web content.

• Text and metadata of HTML pages

  • Newspaper-extract news, articles, and develop content using Python.

  • Html2text-convert HTML into Markdown text.

  • Python-goose-HTML content/Article Extraction Tool.

  • Lassie-a user-friendly web content retrieval tool

  • Micawber-a small library that extracts rich content from the website.

  • Sumy-a module that automatically summarizes text files and HTML webpages

  • Haul-a scalable image crawler.

  • Python-readability-arc90 quick Python interface of readability tool.

  • Scrapely-database for extracting structured data from HTML webpages. Some examples of Web pages and data extraction are provided. scrapely builds a analyzer for all similar Web pages.

• Video

  • Youtube-dl-a small command line program for downloading videos from YouTube.

  • You-get-Python3 video download tool for YouTube and Youku/Niconico.

• Wikipedia

  • WikiTeam-download and save the wikis tool.

WebSocket

The library used for WebSocket.

• Crossbar-open-source application message passing router (WebSocket and WAMP implemented by Python for Autobahn ).

• AutobahnPython-provides the Python Implementation of WebSocket protocol and WAMP Protocol and is open-source.

• WebSocket-for-Python 2 and 3 and PyPy WebSocket client and server library.

DNS resolution

• Dnsyo-check your DNS on more than 1500 DNS servers worldwide.

• Pycares-c-ares interface. C-ares is a c language library for DNS requests and asynchronous name resolution.

Computer Vision

• OpenCV-open-source computer vision library.

• SimpleCV-Brief Introduction to cameras, image processing, feature extraction, format conversion, and highly readable interfaces (based on OpenCV ).

• Mahotas-fast computer image processing algorithm (fully implemented using C ++), which uses a numpy-based array as its data type.

Proxy Server

• Shadowsocks-a fast tunnel proxy that can help you penetrate the firewall (supports TCP and UDP, TFO, multi-user and smooth restart, and target IP blacklist ).

• Tproxy-tproxy is a simple TCP routing proxy (Layer 2). It is configured in Python Based on Gevent.

List of other Python tools

• Awesome-python

• Pycrumbs

• Python-github-projects

• Python_reference

• Pythonidae