Scrapy Crawl 360 site Images

Project Name: Crawl 360 site Pictures

Target url:http://image.so.com

Project Description: Use Scrapy's imagepipeline to crawl pictures of 360 websites

To grab a picture using scrapy, the first step is to define the item first

1 # -*-coding:utf-8-*- 2 Import scrapy 3 4 class Imageitem (scrapy. Item):5     image_urls = scrapy. Field ()

The second step is to open imagepipeline in settings.py and define the file's storage path.

1 item_pipelines = {2     'scrapy.pipelines.images.ImagesPipeline ': 1,3}45'alfred_images' 

The third step is to write the spider.

1 #-*-coding:utf-8-*-2 Importscrapy3 ImportJSON
From.. Items Import Imageitem4 5 classImagesspider (scrapy. Spider):6Name ='Images'7Start_urls ='http://image.so.com/zj?ch=home&sn=90&listtype=new&temp=1'8 9     defParse (self, response):Tenpics = Json.loads (response.body)#because the JSON data is returned, all of this is one more parsing step . One          forIinchpics['List']: Aitem =Imageitem () -item['Image_urls'] = [i["Qhimg_url"]]#Note here that the URL of the picture must be placed in a list, otherwise an exception will occur -             yieldItem

A few points to note:

1. It is best to define Image_urls = Field () in item.py

2. Remember to open imagepipeline in settings and define the image storage path

3. The URL of the picture must be placed in the list, must be placed in the list, must be placed in the list.

Scrapy Crawl 360 site Images