Python Instant web crawler project: Definition of content Extractor (Python2.7 version)

1. Project background

650) this.width=650; "id=" aimg_1219 "src=" http://www.gooseeker.com/doc/data/attachment/forum/201608/05/ 095427wsei82tz6swg2rsg.jpg "class=" Zoom "width=" "height=" 342 "style=" margin-top:10px; "alt=" 095427wsei82tz6swg2rsg.jpg "/>

In the Python instant web crawler Project Launch Note We discuss a number: programmers waste too much time on debugging content extraction rules (see), so we launched this project, freeing programmers from cumbersome debugging rules and putting them into higher-end data processing.

This project has been a great concern since the introduction of open source, we can be developed on the basis of off-the-shelf source. However, Python3 and Python2 are different, the Python instant web crawler project: the definition of content extractor, the source code cannot be used under Python2.7, this article will publish a Python2.7 content extractor.

2. Solution
In order to solve this problem, we isolate the extractor which affects the universality and efficiency, and describe the following data processing flowchart:

650) this.width=650; "id=" aimg_1216 "src=" http://www.gooseeker.com/doc/data/attachment/forum/201608/03/ 222052q9trrar86traqv25.png "class=" Zoom "width=" 487 "style=" margin-top:10px; "alt=" 222052q9trrar86traqv25.png "/ >

The "Pluggable extractor" in the figure must be highly modular, so the key interfaces are:

• Normalized input: As input to standard HTML DOM objects

• Standardized content extraction: extracting Web content using standard XSLT templates

• Normalized output: Output extracted from a Web page in a standard XML format

• Explicit extractor plug-in interface: Extractor is a well-defined class that interacts with the Crawler engine module through class methods
  

3. Extractor code
pluggable extractor is the core component of the instant web crawler project, defined as a class: Gsextractor
        The source code file for python2.7 and its explanatory documentation can be obtained from the   GitHub   Downloads

The usage pattern is this:

1. Instantiate a Gsextractor object

2. Setting an XSLT extractor for this object is equivalent to configuring this object (using the three-class Setxxx () method).

3. Input the HTML DOM to it to get the XML output (using the Extract () method)

Here is the source code for this Gsextractor class (for Python2.7)

#!/usr/bin/python# -*- coding: utf-8 -*-#  Module Name: gooseeker_py2#  class name:  gsextractor# version: 2.0#  Adaptation python version: 2.7#  description:  html content extractor #  feature:  Use XSLT as a template to quickly extract content from Html dom. # released by  collection (http://www.gooseeker.com)  on may 18, 2016# github:  https://github.com/fullerhua/jisou/core/gooseeker_py2.pyfrom urllib2 import urlopenfrom  urllib import quotefrom lxml import etreeimport timeclass gsextractor (object):     def _init_ (self):         self.xslt =  ""     #  read xslt    def  from File Setxsltfromfile (Self , xsltfilepath):         file =  open (xsltfilepath ,  ' R ')         try:      &nbsP;      self.xslt = file.read ()          finally:            file.close ()      #  get Xslt    def setxsltfrommem (SELF , XSLTSTR) from a string:         self.xslt = xsltStr    #  Xslt    def setxsltfromapi (self , apikey ) obtained through the Gooseeker api interface.  theme, middle=none, bname=none):         apiurl =   "http://www.gooseeker.com/api/getextractor?key=" + apikey + "&theme=" +quote (theme)          if  (middle):             apiurl = apiurl +  "&middle=" +quote (middle)          if  (bname):            apiurl =  apiurl +  "&bname=" +quote (bname)         apiconn =  urlopen (Apiurl)         self.xslt = apiconn.read ()     #  returns the current xslt    def getxslt (self):         return self.xslt    #  extraction method, the entry parameter is a Html dom object, and the return is the extraction result     def extract (self , html):         xslt_root = etree. XML (SELF.XSLT)         transform = etree. XSLT (xslt_root)         result_tree = transform (HTML)          return result_tree

4. Usage example
Below is an example program that demonstrates how to use the Gsextractor class to extract a watercress discussion group topic. This example has the following characteristics:

• The contents of the extractor are obtained through the API on the Gooseeker platform

• Save the result file to the current folder
  
  

Here is the source code, which can be downloaded from github

# _*_coding:utf8_*_# douban_py2.py#  Crawl The Watercress Group Discussion topic # python version: 2.7from lxml  import etreefrom gooseeker_py2 import gsextractorfrom selenium import  Webdriverimport timeclass phantomspider:    def getcontent (Self, url):         browser = webdriver. Phantomjs (executable_path= ' C:\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe ')          browser.get (URL)         time.sleep (3)          html = browser.execute_script ("return  Document.documentElement.outerHTML ")         output = etree. HTML (HTML)         return output    def  Savecontent (self, filepath, content):      &nbsP;  file_obj = open (filepath,  ' W ')          File_obj.write (content)         file_obj.close () doubanExtra =  gsextractor ()    #  The following sentence calls Gooseeker's API to set the XSLT crawl rule #  The first parameter is App key, Please go to Gooseeker Member Center to apply for #  the second parameter is the rule name, which is ms  generated by Gooseeker's graphical tool:  plots Doubanextra.setxsltfromapi (" ffd5273e213036d812ea298922e2627b " , " Watercress Group Discussion topic ")   url = " https://www.douban.com/ Group/haixiuzu/discussion?start= "Totalpages = 5doubanspider = phantomspider () print (" Crawl start ") For pagenumber in range (1 , totalpages):     currenturl =  url + str ((pagenumber-1) *25)     print ("Crawling",  currenturl)      content = doubanspider.getcontent (Currenturl)     outputxml =  doubanextra.extract (content)     outputfile =  "Result"  + str (pagenumber)  + ". xml"      Doubanspider.savecontent (Outputfile , str (outputxml)) print ("Crawl End")

The extraction results are as follows:

650) this.width=650; "id=" aimg_1208 "src=" http://www.gooseeker.com/doc/data/attachment/forum/201608/03/ 120504b2e71a1ipzi1prxb.png "class=" Zoom "width=" "style=" margin-top:10px; "alt=" 120504b2e71a1ipzi1prxb.png "/ >

5. Next Read
This article has explained the value and usage of the extractor, but did not say how to generate it, only the rapid generation of extractor to achieve the purpose of saving developers time, this question will be explained in other articles, see the 1-minute quick generate XSLT template for Web content extraction

6. Set search Gooseeker source code Download
1. Gooseeker Open source Python web crawler github source

7. History of Document Modification
the Content Extractor class under 2016-08-05:v1.0,python2.7 is first published

This article from "Fullerhua blog" blog, declined reprint!

Python Instant web crawler project: Definition of content Extractor (Python2.7 version)