(8) How to do the crawler scrapy under distributed-download (source stacking)

Reprint main Note Source: http://www.cnblogs.com/codefish/p/4968260.html

In the crawler, we encounter more demand is the file download and image download, in other languages or frameworks, we may be in the data filtering, and then asynchronously use the file download class to achieve the purpose, scrapy framework itself has implemented files and pictures downloaded files, quite convenient, as long as a few lines of code, You can easily get the download done. Below I will show how to use Scrapy Download Watercress album home page content.

Introduction of Advantages:

1) automatic weight-off

2) asynchronous operation, no blocking

3) can generate thumbnails of specified dimensions

4) Calculate Expiration time

5) format Conversion

Encoding process:

One, define the item

#-*-coding:utf-8-*-#Define Here the models for your scraped items##See documentation in:#http://doc.scrapy.org/en/latest/topics/items.htmlImportscrapy fromScrapyImportItem,fieldclassDoubanimgsitem (scrapy. Item):#Define the fields for your item here is like:    #name = Scrapy. Field ()Image_urls =Field () images=Field () image_paths=Field ()Pass

Two, define Spider

#Coding=utf-8 fromScrapy.spidersImportSpiderImportRe fromDouban_imgs.itemsImportDoubanimgsitem fromScrapy.http.requestImportRequest#attention to the encoding of Info,otherwise raise errorImportsysreload (SYS) sys.setdefaultencoding ('UTF8')classDownload_douban (Spider): Name='Download_douban'    def __init__(Self, url='152686895', *args, * *Kwargs): Self.allowed_domains= ['douban.com'] Self.start_urls= [                'http://www.douban.com/photos/album/%s/'%(URL)]#Call the Father base functionSelf.url =URL super (Download_douban, self).__init__(*args, * *Kwargs)defParse (self, response):""": type Response:response infomation"""List_imgs= Response.xpath ('//div[@class = "Photolst clearfix"]//img/@src'). Extract ()ifList_imgs:item=Doubanimgsitem () item['Image_urls'] =List_imgsyieldItem

Third, define Piepline

#-*-coding:utf-8-*-#Define your item pipelines here##Don ' t forget to add your pipeline to the Item_pipelines setting#see:http://doc.scrapy.org/en/latest/topics/item-pipeline.html fromScrapy.pipelines.imagesImportImagespipeline fromScrapy.exceptionsImportDropitem fromScrapyImportRequest fromScrapyImportLogclassDoubanimgspipeline (object):defProcess_item (self, item, spider):returnItemclassDoubanimgdownloadpieline (imagespipeline):defget_media_requests (self,item,info): forImage_urlinchitem['Image_urls']:            yieldRequest (Image_url)defitem_completed (self, results, item, info): Image_paths= [x['Path'] forOK, XinchResultsifOK]if  notimage_paths:RaiseDropitem ("Item contains no images") item['image_paths'] =image_pathsreturnItem

Four, define setting.py, enable item processor

# Configure Item Pipelines # See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html Item_pipelines = {   'douban_imgs.pipelines.DoubanImgDownloadPieline': 300 ,} Images_store='c:\\doubanimgs'= 90

Operating effect:

Reprint main Note Source: http://www.cnblogs.com/codefish/p/4968260.html

If Scrapy or Reptile series is helpful to you, please recommend, I will update more crawler series later

(8) How to do the crawler scrapy under distributed-download (source stacking)