Download files and Images

Download files and Images

Scrapy provides a reusableitem pipelines. ThesepipelineSome common methods and structures are calledmedia pipeline. Generally, you will useFiles PiplineOrImages Pipeline.

Why use scrapyBuilt-in File Download Method

1. Avoid downloading data that has been downloaded recently.
2. You can easily specify the file storage path.
3. You can convert the downloaded image to a common format. For example, PNG or JPG.
4. You can easily produce thumbnails.
5. The image width and height can be easily detected to ensure that they meet the minimum limit.
6. Asynchronous download with high efficiency.

Download file File Pipeline:

When usingFiles PiplineTo download an object, follow these steps:

1. Define oneItemAnd thenitemDefine two properties, respectivelyfile_urlsAndfiles.file_urlsIs used to store the URL link of the file to be downloaded, and a list is required.
2. After the file is downloaded, the downloaded information is stored initemOffilesAttribute. Such as the download path, download URL, and file verification code.
3. In the configuration filesettings.pyConfiguringFILES_STOREThis configuration is used to set the File Download path.
4. Startpipeline: InITEM_PIPELINESSet inscrapy.pipelines.files.FilesPipline:1.

Download Image Images Pipline:

When usingImages PipelineTo download an object, follow these steps:

1. Define oneItemAnd thenitemDefine two properties, respectivelyimage_urlsAndimages.image_urlsIs used to store the URL link of the file to be downloaded, and a list is required.
2. After the file is downloaded, the downloaded information is stored initemOfimagesAttribute. Such as the download path, download URL, and file verification code.
3. In the configuration filesettings.pyConfiguringIMAGES_STOREThis configuration is used to set the download path of the image.
4. Startpipeline: InITEM_PIPELINESSet inscrapy.pipelines.images.ImagesPipline:1

Download files and Images