Installation of Sesame Http:scrapy-splash

Scrapy-splash is a tool that supports JavaScript rendering in Scrapy, and this section describes how it is installed.

The installation of Scrapy-splash is divided into two parts. One is the installation of the splash service, specifically through Docker, after installation, a splash service is launched, and we can implement the JavaScript page loading through its interface. The other is the installation of the Scrapy-splash Python library, which can be used to scrapy the splash service after installation.

1. RELATED LINKS

• Github:https://github.com/scrapy-plugins/scrapy-splash
• Pypi:https://pypi.python.org/pypi/scrapy-splash
• Instruction for use: https://github.com/scrapy-plugins/scrapy-splash#configuration
• Splash Official Document: Http://splash.readthedocs.io

2. Installing Splash

Scrapy-splash uses the Splash HTTP API for page rendering, so we need to install splash to provide rendering services. This is done by Docker installation, before making sure that Docker is properly installed.

The installation commands are as follows:

8050:8050 Scrapinghub/splash

After the installation is complete, there are similar output results:

 .- --Geneva  ,: -: -+0000[-] Log opened. .- --Geneva  ,: -:28.447291[-] Splash version:3.0 .- --Geneva  ,: -:28.452698[-] Qt5.9.1, PyQt5.9Webkit602.1Sip4.19.3, Twisted16.1.1Lua5.2 .- --Geneva  ,: -:28.453120[-] Python3.5.2(default, Nov -  ., -: to: at) [GCC5.4.0 20160609] .- --Geneva  ,: -:28.453676[-] Open Files Limit:1048576 .- --Geneva  ,: -:28.454258[-] Can'T bump open files limit .- --Geneva  ,: -:28.571306[-] Xvfb isStarted: ['XVFB',': 1599197258','-screen','0','1024x768x24','-nolisten','TCP']qstandardpaths:xdg_runtime_dir notSet, defaulting to'/tmp/runtime-root' .- --Geneva  ,: -:29.041973[-] Proxy profiles Support isEnabled, proxy profiles path:/etc/splash/proxy-Profiles .- --Geneva  ,: -:29.315445[-] Verbosity=1 .- --Geneva  ,: -:29.315629[-] Slots= - .- --Geneva  ,: -:29.315712[-] Argument_cache_max_entries= - .- --Geneva  ,: -:29.316564[-] Web ui:enabled, lua:enabled (sandbox:enabled) .- --Geneva  ,: -:29.317614[-] Site starting on8050 .- --Geneva  ,: -:29.317801[-] Starting Factory <twisted.web.server.siteObjectAt0x7ffaa4a98cf8>

This proves that the splash is already running on port 8050. When we open http://localhost:8050, we can see the home page of Splash, as shown in 1-80.

Figure 1 Running the page

Of course, splash can also be installed directly on the remote server. We run splash on the server as a daemon, with the following commands:

8050:8050 Scrapinghub/splash

Here are the -d parameters, which represent the Docker container running in the daemon so that the splash service is not terminated after the remote server connection is interrupted.

3. Installation of Scrapy-splash

After successfully installing splash, then install its Python library with the following command:

PIP3 Install Scrapy-splash

Once the command is complete, the library will be installed successfully.

Installation of Sesame Http:scrapy-splash