Experience in installing Scrapy from scratch, and experience in installing Scrapy from scratch

Experience in installing Scrapy from scratch, and experience in installing Scrapy from scratch

Scrapy is a python-based Web Crawler framework that crawls information from the network. It is a good way to obtain data. So I want to install it.

Go to its official website and go to the Installation Introduction page.

Https://docs.scrapy.org/en/latest/intro/install.html

There are three installation methods: one is from pip, the other is from source code, and the other is from conda

Based on previous knowledge, pip is already a package management tool integrated in python. It is simple and clear, and ignores a sentence on the official website introducing the interface.

Note that sometimes this may require solving compilation issues for some Scrapy dependencies depending on your operating system

Many errors are reported in the compilation phase.

Then I gave up. Compiling from the source code, like pip, is also a bunch of compilation errors.

No way, go to the conda and download a miniconda, more than 60 MB. The results of a careful study are no longer satisfactory.

Python may also notice that its package needs to be compiled after it is downloaded. If it is compiled, it depends on the environment configuration of its own OS, which often causes errors.

Miniconda is an integrated environment with python installed. It is equivalent to downloading and installing miniconda, that is, downloading the basic python core program, then, you can use the conda command to download the conda compiled package for Function Extension. That is, the scrapy package and its dependent lxml, twisted, and other compiled half-dead packages are already compiled. Then download and use it directly.

Https://conda.io/docs/install/quick.html

Https://conda.io/miniconda.html