python增量爬蟲pyspider

標籤：html   com   比較   知識庫   並且   主機   null   bash   headers   

1.為了能夠將爬取到的資料存入本機資料庫，現在本地建立一個MySQL資料庫example，然後 
在資料庫中建立一張表格test，樣本如下：

DROP TABLE IF EXISTS `test`;CREATE TABLE `douban_db` (  `id` int(11) NOT NULL AUTO_INCREMENT,  `url` varchar(20) NOT NULL,  `direct`  varchar(30),  `performer`  date,  `type`  varchar(30),  `district` varchar(20) NOT NULL,  `language`  varchar(30),  `date`  varchar(30),  `time`  varchar(30),  `alias` varchar(20) NOT NULL,  `score`  varchar(30),  `comments`  varchar(300),  `scenario`  varchar(300),  `IMDb`  varchar(30),  PRIMARY KEY (`id`)) ENGINE=MyISAM  DEFAULT CHARSET=utf8;

 

2.如果使用開源架構pyspider來進行爬蟲的話，預設情況下，會把爬取到的結果存放到result.db這個sqilite資料庫中，但是為了方便操作，我們將結果存放到mysql中。接下 
來要做的一個操作就是重寫on_result方法，執行個體化調用我們自己實現的SQL方法，具體 
執行個體如下：

#!/usr/bin/env python# -*- encoding: utf-8 -*-# Created on 2015-03-20 09:46:20# Project: fly_spiderimport refrom pyspider.database.mysql.mysqldb import SQLfrom pyspider.libs.base_handler import *class Handler(BaseHandler):    headers= {                          "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",    "Accept-Encoding":"gzip, deflate, sdch",    "Accept-Language":"zh-CN,zh;q=0.8",    "Cache-Control":"max-age=0",    "Connection":"keep-alive",    "User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36"    }    crawl_config = {        "headers" : headers,        "timeout" : 100    }    @every(minutes=24 * 60)    def on_start(self):        self.crawl(‘http://movie.douban.com/tag/‘, callback=self.index_page)    @config(age=10 * 24 * 60 * 60)    def index_page(self, response):        for each in response.doc(‘a[href^="http"]‘).items():            if re.match("http://movie.douban.com/tag/\w+", each.attr.href, re.U):                self.crawl(each.attr.href, callback=self.list_page)      @config(age=10*24*60*60, priority=2)                    def list_page(self, response):        for each in response.doc(‘html > body > div#wrapper > div#content > div.grid-16-8.clearfix > div.article > div > table tr.item > td > div.pl2 > a‘).items():            self.crawl(each.attr.href, priority=9, callback=self.detail_page)      @config(priority=3)    def detail_page(self, response):        return {            "url": response.url,            "title": response.doc(‘html > body > #wrapper > #content > h1 > span‘).text(),            "direct": ",".join(x.text() for x in response.doc(‘a[rel="v:directedBy"]‘).items()),            "performer": ",".join(x.text() for x in response.doc(‘a[rel="v:starring"]‘).items()),            "type": ",".join(x.text() for x in response.doc(‘span[property="v:genre"]‘).items()),#            "district":  "".join(x.text() for x in response.doc(‘a[rel="v:starring"]‘).items()),#            "language":  "".join(x.text() for x in response.doc(‘a[rel="v:starring"]‘).items()),            "date":  ",".join(x.text() for x in response.doc(‘span[property="v:initialReleaseDate"]‘).items()),            "time":  ",".join(x.text() for x in response.doc(‘span[property="v:runtime"]‘).items()),#            "alias":  "".join(x.text() for x in response.doc(‘a[rel="v:starring"]‘).items()),            "score": response.doc(‘.rating_num‘).text(),            "comments": response.doc(‘html > body > div#wrapper > div#content > div.grid-16-8.clearfix > div.article > div#comments-section > div.mod-hd > h2 > i‘).text(),            "scenario": response.doc(‘html > body > div#wrapper > div#content > div.grid-16-8.clearfix > div.article > div.related-info > div#link-report.indent‘).text(),            "IMDb":  "".join(x.text() for x in response.doc(‘span[href]‘).items()),            }    def on_result(self, result):        if not result or not result[‘title‘]:            return        sql = SQL()        sql.replace(‘douban_db‘,**result)    

 

關於上面這段代碼，有下面幾點需要說明的： 
a. 為了避免伺服器判斷出用戶端在進行爬蟲操作，從而禁止ip訪問（具體表現為出現403禁止訪問），我們需要在發出請求的時候加上一個http頭，偽裝成使用瀏覽器訪問，具體用法如下：

    headers= {                          "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",    "Accept-Encoding":"gzip, deflate, sdch",    "Accept-Language":"zh-CN,zh;q=0.8",    "Cache-Control":"max-age=0",    "Connection":"keep-alive",    "User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36"    }    crawl_config = {        "headers" : headers,        "timeout" : 100    }

 

b. @every(minutes=24 * 60)表示每天執行一次 
@config(age=10 * 24 * 60 * 60)表示資料10天后就到期了

c. 接下來是一個比較重要的地方，重寫on_result方法，相當於實現了一個多態，程式在最後返回時，會執行on_result方法，預設的情況下，on_result是將資料刷入sqlite中，但是如果我們需要將資料插入mysql中，就需要重寫on_result方法，具體使用如下：

    def on_result(self, result):        if not result or not result[‘title‘]:            return        sql = SQL()        sql.replace(‘test‘,**result)    

注意這裡的if not result or not result[‘title’]:這句判斷很重要，不然的會會報錯，提示result是未定義類型的。

3.在上面的額代碼中，提到了執行個體化調用我們自己實現的SQL方法，並且引用了from pyspider.database.mysql.mysqldb import SQL這個庫檔案，那麼就必須在這個目錄下實現這個庫，具體如下： 
把下面內容文文放到pyspider/pyspider/database/mysql/目錄下命名為mysqldb.py

from six import itervaluesimport mysql.connectorfrom datetime import date, datetime, timedeltaclass SQL:        username = ‘root‘  #資料庫使用者名稱        password = ‘root‘  #資料庫密碼        database = ‘test‘  #資料庫        host = ‘172.30.25.231‘  #資料庫主機地址        connection = ‘‘        connect = True    placeholder = ‘%s‘        def __init__(self):                if self.connect:                        SQL.connect(self)    def escape(self,string):        return ‘`%s`‘ % string        def connect(self):            config = {                ‘user‘:SQL.username,                ‘password‘:SQL.password,                ‘host‘:SQL.host            }            if SQL.database != None:                config[‘database‘] = SQL.database            try:                cnx = mysql.connector.connect(**config)                SQL.connection = cnx                return True            except mysql.connector.Error as err:            if (err.errno == errorcode.ER_ACCESS_DENIED_ERROR):                print "The credentials you provided are not correct."            elif (err.errno == errorcode.ER_BAD_DB_ERROR):                print "The database you provided does not exist."            else:                print "Something went wrong: " , err            return False    def replace(self,tablename=None,**values):        if SQL.connection == ‘‘:                    print "Please connect first"                    return False                tablename = self.escape(tablename )                if values:                        _keys = ", ".join(self.escape(k) for k in values)                        _values = ", ".join([self.placeholder, ] * len(values))                        sql_query = "REPLACE INTO %s (%s) VALUES (%s)" % (tablename, _keys, _values)                else:                        sql_query = "REPLACE INTO %s DEFAULT VALUES" % tablename        cur = SQL.connection.cursor()                try:                    if values:                            cur.execute(sql_query, list(itervalues(values)))                    else:                            cur.execute(sql_query)                    SQL.connection.commit()                    return True                except mysql.connector.Error as err:                    print ("An error occured: {}".format(err))                    return False

 

學習文檔：http://blog.binux.me/2015/01/pyspider-tutorial-level-1-html-and-css-selector/ 
測試環境：http://demo.pyspider.org/

python增量爬蟲pyspider