Use the Python Scrapy framework to crawl the beauty chart in ten minutes

Scrapy is a lightweight, simple, and easy to use method written in Python. Scrapy can be used to conveniently collect online data. it has done a lot of work for us, rather than making great efforts to develop it. This article describes how to use the Python Scrapy framework to crawl the beauty chart in ten minutes. For more information, see Scrapy. it is a lightweight, simple, and easy-to-use method written in Python. Scrapy can be used to conveniently collect online data. it has done a lot of work for us, rather than making great efforts to develop it. This article describes how to use the Python Scrapy framework to crawl the beauty chart in ten minutes. For more information, see

Introduction

Scrapy is a python crawler framework with rich functions and convenient use. Scrapy can be used to quickly develop a simple crawler. an Official simple example is enough to prove its strength:

Quick development

The next 10-minute countdown starts:

Of course, before you start, you can first read the scrapy getting started article "writing python crawlers using the Scrapy framework ".

1. initialize the project

scrapy startproject mztcd mztscrapy genspider meizitu meizitu.com

2. add the spider code:

Define scrapy. Item and add image_urls and images to prepare for downloading the image.

Modify start_urls as the initial page, add parse to process the list page, and add parse_item to process the project page.

3. modify the configuration file:

DOWNLOAD_DELAY = 1 # add the download delay configuration ITEM_PIPELINES = {'scrapy. pipelines. images. ImagesPipeline ': 1} # add an image download pipelineIMAGES_STORE ='. '# set the image storage directory

4. run the project:

scrapy crawl meizitu

See, project run

For more articles about using the Python Scrapy framework to crawl the beauty chart in ten minutes, please follow the PHP Chinese network!