Deployment and Application of splash in scrapy

Because we need to capture part of the webpage data rendered by JavaScript, the request in scrapy returns the webpage code that has not been rendered,

Therefore, we need to use the splashrequest in the scrapy splash for operations.

First, install scrapy-splash:

1. Use Pip to install scrapy-splash

PIP install scrapy-splash

2. Install docker

Docker installation is cumbersome. First, you need to go to the official website to download the docker tool. Because it is Windows 7, you need to download the dockertoolbox and install it manually.

Then, the following occurs during the installation process: (this figure is found online)

In this case, you need to go to GitHub to download the boot2docker-18.03.0-ce file and put it in the yellow line path (each person's path may be different ),

The Red Arrow is https://github.com/boot2docker/boot2docker/releases/tag/v18.03.0-ce,

Then, place the downloaded file to the specified address and wait for running (For A Long Time)

Run docker Quickstart terminal after installation,

Enter docker pull scrapinghub/splash

Then input docker run-P 8050: 8050 scrapinghub/splash

In this way, docker is enabled.

Then you can start to use the splashrequest in scrapy-splash in Python.

3. Set the setting file in Python

Splash_url = 'HTTP: // 192.168.99.100: 100)

Add the splash middleware and specify the priority:

Downloader_middlewares = {
'Scrapy _ splash. splashcookiesmiddleware ': 723,
'Scrapy _ splash. splashmiddleware ': 725,
'Scrapy. downloadermiddlewares. httpcompression. httpcompressionmiddleware ': 810,
}

 

Set splash's own deduplication filter:
Dupefilter_class = 'scrapy _ splash. splashawaredupefilter'

Cache backend storage media:
Httpcache_storage = 'scrapy _ splash. splashawarefscachestore' # the preceding two must be added.

 

4. Use splashrequest:

Note the following:

Yield splashrequest (url = news_url, callback = self. down_load, argS = {"wait": 3 })

The above is usedAn example of splashrequest, which must be addedARGs = {"wait": 3}How long will this logo wait before sending webpage information.

Deployment and Application of splash in scrapy