scrapy架構初級

標籤：callback   request   應用   download   程式   直接下載   false   next   ace   

scrapy入門教程：https://scrapy-chs.readthedocs.io/zh_CN/0.24/intro/tutorial.html一、安裝python模組網站，應用檔案放置在scrips下，whl：https://www.lfd.uci.edu/~gohlke/pythonlibs/ Scrapy架構依賴 Twistid需要再上邊網站下載，放置scrips下；??pip install C:\python\Anaconda3\Twisted-18.7.0-cp36-cp36m-win_amd64.whl??pip install scrapy 二、建立Scrapy項目1.由於pychram沒有Integration Environment，需要執行命令建立，執行完，用pychram選擇新視窗開啟；??????????scrapy startproject  projectname 2.建立爬蟲檔案執行命令如下：命令部分??????????????????檔案名稱  爬取得網站scrapy genspider baidu baidu.comscrapy genspider -t crawl baidu baidu.com 3設定檔修改：settings.py檔案

USER_AGENT = ‘Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.79 Safari/537.36 Maxthon/5.2.3.6000‘

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

DOWNLOAD_DELAY = 3

ITEM_PIPELINES = {
   ‘xiaoshuo_pc.pipelines.XiaoshuoPcPipeline‘: 300,
}

 4運行程式：scrapy crawl name（變數值）scrapy crawl name -o book.json（輸出到檔案{json、xml、csv}）scrapy crawl name -o book.json -t json(-t 代表格式輸出，一般忽略)

**第一次啟動並執行時候，我遇到no module named win32API錯誤，這是因為Python沒有內建訪問windows系統API的庫的，需要下載第三方庫。庫的名稱叫pywin32，可以從網上直接下載，下載連結：http://sourceforge.net/projects/pywin32/files%2Fpywin32/ （下載適合你的Python版本）下載後放置到scripts目錄下雙機運行，即可（（或者pip install pypiwin32）） 

三、小說擷取範例程式碼：建立入口執行檔案main.py

from scrapy.cmdline import execute
execute("scrapy crawl zol".split()) # zol為zol檔案中的變數定義的名

class ShiqikSpider(scrapy.Spider):
    name = ‘shiqik‘
    allowed_domains = [‘17k.com‘]
    start_urls = [‘https://www.81zw.us/book/1379/6970209.html‘]

    def parse(self, response):
        title=response.xpath(‘//div[@class="bookname"]/h1/text()‘).extract_first()
        content=‘‘.join(response.xpath(‘//div[@id="content"]/text()‘).extract()).replace(‘   ‘,‘\n‘)
        yield {"title":title,"content":content}
        next_page=response.xpath(‘//div[@class="bottem2"]/a[3]/@href‘).extract_first()
        if next_page.find(".html")!=-1:
            print("繼續下一個url")
            new_url=response.urljoin(next_page)
            yield scrapy.Request(new_url,callback=self.parse,dont_filter=True)

  四、小說擷取範例程式碼：   

class BayizhongwenSpider(CrawlSpider):
    name = ‘bayizhongwen‘
    allowed_domains = [‘81zw.us‘]
    # start_urls = [‘https://www.81zw.us/book/1215/863759.html‘]
    start_urls = [‘https://www.81zw.us/book/1215‘]

    rules = (
        Rule(LinkExtractor(restrict_xpaths=r‘//dl/dd[2]/a‘), callback=‘parse_item‘, follow=True),
        Rule(LinkExtractor(restrict_xpaths=r‘//div[@class="bottem1"]/a[3]‘), callback=‘parse_item‘, follow=True),
    )
    def parse_item(self, response):
        title=response.xpath(‘//div[@class="bookname"]/h1/text()‘).extract_first()
        content=‘‘.join(response.xpath(‘//div[@id="content"]/text()‘).extract()).replace(‘   ‘,‘\n‘)
        print({"title":title,"content":content})
        yield {"title":title,"content":content}

    一、建立項目 (venv) C:\Users\noc\PycharmProjects>scrapy startproject tupian 二、建立app (venv) C:\Users\noc\PycharmProjects\tupian>scrapy genspider zol zol.com.cn 三、修改配置資訊settings.py檔案：

# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36‘

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

DOWNLOAD_DELAY = 3

 

# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   # ‘tupian.pipelines.TupianPipeline‘: 300,
    ‘scrapy.contrib.pipeline.images.ImagesPipeline‘: 300,
}
# 增加圖片存放目錄
IMAGES_STORE=‘e:/img‘

  四、建立入口執行檔案start.py

from scrapy.cmdline import execute
execute("scrapy crawl zol".split()) # zol為zol檔案中的變數定義的名

五、主檔案代碼：

import scrapy


class ZolSpider(scrapy.Spider):
    name = ‘zol‘
    allowed_domains = [‘zol.com.cn‘]
    start_urls = [‘http://desk.zol.com.cn/bizhi/7239_89590_2.html‘]  # 爬取圖片頁面的地址

    def parse(self, response):
        image_url = response.xpath(‘//img[@id="bigImg"]/@src‘).extract()  # 爬取第一張圖片的地址
        image_name = response.xpath(‘string(//h3)‘).extract_first()  # 爬取圖片名稱
        yield {"image_url": image_url, "image_name": image_name}  # 推送
        next_page = response.xpath(‘//a[@id="pageNext"]/@href‘).extract_first()  # 爬取圖片下一張按鈕的地址
        if next_page.find(‘.html‘) != -1:  # 判斷最後一張圖片地址如果不包含.html
            yield scrapy.Request(response.urljoin(next_page), callback=self.parse)

六、middlewares檔案

from tupian.settings import USER_AGENT
from random import choice
from fake_useragent import UserAgent


# User-Agent設定
class UserAgentDownloaderMiddleware(object):
    def process_request(self, request, spider):
        # if self.user_agent:
        # request.headers.setdefault(b‘User-Agent‘,choice(USER_AGENT))
        request.headers.setdefault(b‘User-Agent‘, UserAgent().random)

# 代理設定
class ProxyMiddleware(object):
    def process_request(self, request, spider):
        # request.meta[‘proxy‘]=‘http://ip:port‘
        request.meta[‘proxy‘]=‘http://124.235.145.79:80‘
        # request.meta[‘proxy‘]=‘http://user:[email protected]:port‘
        # request.meta[‘proxy‘]=‘http://398707160:[email protected]:16816‘

                  

scrapy架構初級