Python Instant web crawler project: Definition of content Extractor (Python2.7 version)

1. Project background

In the Python instant web crawler Project Launch Note We discuss a number: programmers waste too much time on debugging content extraction rules (see), so we launched this project, freeing programmers from cumbersome debugging rules and putting them into higher-end data processing.

This project has been a great concern since the introduction of open source, we can be developed on the basis of off-the-shelf source. However, Python3 and Python2 are different, the Python instant web crawler project: the definition of content extractor, the source code cannot be used under Python2.7, this article will publish a Python2.7 content extractor.

2. Solution

In order to solve this problem, we isolate the extractor which affects the universality and efficiency, and describe the following data processing flowchart:

The "Pluggable extractor" in the figure must be highly modular, so the key interfaces are:

• Normalized input: As input to standard HTML DOM objects
• Standardized content extraction: extracting Web content using standard XSLT templates
• Normalized output: Output extracted from a Web page in a standard XML format
• Explicit extractor plug-in interface: Extractor is a well-defined class that interacts with the Crawler engine module through class methods

3. Extractor code

The pluggable extractor is the core component of the instant web crawler project, defined as a class: Gsextractor
for python2.7 source code files and their explanatory documentation, please download from GitHub

The usage pattern is this:

• Instantiate a Gsextractor object
• Setting an XSLT extractor for this object is equivalent to configuring this object (using the three-class Setxxx () method).
• Input the HTML DOM to it to get the XML output (using the Extract () method)
  

Here is the source code for this Gsextractor class (for Python2.7)

4. Usage examples

Here is an example program that shows how to use the Gsextractor class to extract a watercress discussion group topic. This example has the following characteristics:

• The contents of the extractor are obtained through the API on the Gooseeker platform
• Save the result file to the current folder
  

Here is the source code, which can be downloaded from github

The extraction results are as follows:

5. Next Read

This article has explained the value and usage of the extractor, but did not say how to generate it, only the rapid generation of extractor to achieve the purpose of saving developers time, this question will be explained in other articles, see the 1-minute quick generate XSLT template for Web content extraction

6. Set Search Gooseeker source code download

1. Gooseeker Open source Python web crawler GitHub source

7. History of Document Modification
The Content Extractor class under 2016-08-05:v1.0,python2.7 is first published

Python Instant web crawler project: Definition of content Extractor (Python2.7 version)