Scrapy installation -------- Windows, linux, mac and other operating platforms, windows Operating linux tools

Scrapy installation -------- Windows, linux, mac and other operating platforms, windows Operating linux tools
Scrapy Installation

Scrapy can be installed in multiple ways. It supports Python or later, or Python or later. The following describes the scrapy Installation Process in the py3 environment.

Scrapy depends on a large number of databases. It must at least rely on the Twisted 14.0, lxml 3.4, and pyOpenSSL 0.14 libraries. The environments on different platforms are different. Therefore, before installation, make sure that you have installed some basic libraries, especially Windows.

I. Anaconda

This method is a relatively simple method for installing scrapy (especially for Windows). You can use this method for installation. You can also select the installation method of the special platform in the following article.

Anaconda is a Python release version that contains common data science libraries. If it is not installed, download the package installation for the platform on the official website https://www.continuum.io/downloads.

If you have already installed scrapy, run the conda command to install scrapy.

The installation is as follows:

Open Anaconda's Anaconda Prompt input first

 conda install Scrapy

Indicates that the installation is successful:

　　

Ii. Install lxml in Windows1.

The best installation method is through the wheel file to install, http://www.lfd.uci.edu /~ Gohlke/pythonlibs/. This website is really a good news for windows users. Basically, it is included in the python library and called it the python universal library website. Find the lxml-related file (ctrl + F) from the website. For Python and windows 64-bit systems, find the lxml 20173.7.2 release cp35 release cp35m release win_amd64.whl file to download, install the SDK using pip.

　　

After the download, run the following command to install:

Pip3 install wheelpip3 install lxml release 3.7.2 release cp35 release cp35m release win_amd64.whl # If the version is python2, change pip3 to pip2

You can complete lxml installation, and replace the file name with other files.

2. Install pyOpenSSL

Download the wheel file from the official website at https://pypi.python.org/pypi/pyopenssl#downloads. download the. whl file and install it.

pip3 install pyOpenSSL-16.2.0-py2.py3-none-any.whl

3. Install Twisted

Similarly, download the http://www.lfd.uci.edu from the python omnipotent website /~ Install gohlke/pythonlibs/# twisted. For python3.5 and windows 64-bit systems, find Twisted ipv17.1.0 255.cp35 255.cp35m 255.win_amd64.whl.
Download this file and install it through pip.

pip3 install Twisted‑17.1.0‑cp35‑cp35m‑win_amd64.whl 

4. Install pywin32

Download the corresponding version of the installation package from the official website https://sourceforge.net/projects/pywin32/files/pywin32/Build%20220.

5. Install Scrapy

Install Scrapy and use pip as follows:

pip3 install Scrapy

Iii. CentOS, RedHat, and Fedora

Install the dependent library to ensure that some necessary class libraries have been installed. Run the following command:

sudo yum groupinstall development toolssudo yum install python34-devel epel-release libxslt-devel libxml2-devel openssldevel
pip3 install Scrapy

Iv. Ubuntu, Debian, and Deepin

Install the dependent library to ensure that some necessary class libraries have been installed. Run the following command:

sudo apt-get install build-essential python3-dev libssl-dev libffi-dev libxml2 libxml2-dev libxslt1-dev zlib1g-dev 

pip3 install Scrapy

V. Mac OS

Dependency library installation. Building Scrapy dependency libraries on Mac requires the C compiler and Development header files. It is generally provided by Xcode and can be installed by running the following command:

xcode-select --install
pip3 install Scrapy

Verification:

After installation, enter scrapy in the command line. If the result is similar to the following, congratulations on your successful installation of scrapy.

　　

6. Common Errors

• Pkg_resources.VersionConflict: (six 1.5.2 (/usr/lib/python3/dist-packages), Requirement. parse ('six> = 1.6.0 '))
  The version of the six package is too low. The six package is a library compatible with Python2 and 3. Upgrade the six package.
  

  sudo pip3 install -U six

• C/_ cffi_backend.c: 15: 17: fatal error: ffi. h: No such file or directory
  The Libffi library is missing. What is libffi? The full name of "FFI" is Foreign Function Interface. It usually refers to the code that can be written in one language to call the code in another language. The Linffi Library only provides the underlying, architecture-related, and complete "FFI ".
  Installation:
  Ubuntu and Debian:Sudo apt-get install build-essential libssl-dev libffi-dev python3-dev
  CentOS, RedHat:Sudo yum install gcc libffi-devel python-devel openssl-devel
  
• ImportError: No module named 'cryptography'
  This is a component that lacks encryption. You can use pip to install it.
  

  sudo pip3 install cryptography

• ImportError: No module named 'Packaging
  The packaging package is missing. It provides the core functions of the Python package. Use pip to install the SDK.
  

  sudo pip3 install packaging

• ImportError: No module named 'appdirs'
  The appdirs package is missing. It is used to determine the file directory. You can use pip to install the package separately.
  

  sudo pip3 install appdirs

Author: Jin Xiao
Source: http://www.cnblogs.com/jinxiao-pu
The copyright of this article is shared by the author and the blog. You are welcome to repost this article, but you must keep this statement without the author's consent and provide a connection to the original article on the article page.