Steps for installing Scrapy in Python3 and python3scrapy

Steps for installing Scrapy in Python3 and python3scrapy

This article describes how to install Scrapy in Python 3:

1. Running platform: Windows
2. Python version: Python3.x
3. IDE: Sublime text3

I. Scrapy Introduction

Scrapy is an application framework written to crawl website data to extract structural data. It can be used in data mining, information processing, and storing historical data among other columns. Scrapy was initially designed for network crawling. Currently, Scrapy has released the promised version of Python3.x.

Why Learning Scrapy? It allows us to better complete crawler tasks. Writing Python crawler programs by ourselves is like fighting alone, and using Scrapy is like having thousands of troops under our hands. Scrapy can get twice the result with half the effort (or even several times. Therefore, it is necessary to learn Scrapy.

Ii. Scrapy Installation

1. directly run the command pip3 install scrapy to find many errors.

1. Failed building wheel for lxml
2. Microsoft Visual C ++ 10.0 is required
3. Failed building twisted
4. Unable to find vcvarsall. bat

The error is shown in:

2. Solution

In http://www.lfd.uci.edu /~ Gohlke/pythonlibs/has many third-party Python libraries compiled for windows. You can download the corresponding Python library.

(1) enter the command python in cmd to view the python version, as shown below:

We can see that my Python version is Python3.5.2-64bit.

(2) login to the http://www.lfd.uci.edu /~ Gohlke/pythonlibs/, Ctrl + F search Lxml, Twisted, Scrapy, download the corresponding version, for example: lxml-3.7.3-cp35-cp35m-win_adm64.whl, indicates that the lxml version is 3.7.3, the corresponding python version is 3.5-64bit. Shows the version I downloaded:

(3) enter the DOS command in cmd and enter the downloaded whl folder. For example, put the three whl files in the Scrapy Folder:

(4) execute the following commands in sequence:

A. pip3 install wheel

B. pip3 install lxml-3.7.3-cp35-cp35m-win_amd64.whl

C. pip3 install Twisted-17.1.0-cp35-cp35m-win_amd64.whl

D. pip3 install Scrapy-1.3.2-py2.py3-none-any.whl

In this way, the installation of Scrapy is complete. Ignore the last two lines for me to upgrade pip. *.*

(5) Srapy has been installed successfully. You must also download pywin32 and find the corresponding version to download. Next step to install it. After the installation is complete, Scrapy can be used normally.

URL: https://sourceforge.net/projects/pywin32/files/pywin32/Build%20220/

Now, we can use Scrapy happily.

Common Errors

1. pkg_resources.VersionConflict: (six 1.5.2 (/usr/lib/python3/dist-packages), Requirement. parse ('six> = 1.6.0 '))

The version of the six package is too low. The six package is a library compatible with Python2 and Python3. Upgrade the six package.

sudo pip3 install -U six

2. c/_ cffi_backend.c: 15: 17: fatal error: ffi. h: No such file or directory

The Libffi library is missing. What is libffi? The full name of "FFI" is Foreign Function Interface. It usually refers to the code that can be written in one language to call the code in another language. The Libffi Library only provides the underlying, architecture-related, and complete "FFI ".

Install the corresponding library.

Ubuntu and Debian:

sudo apt-get install build-essential libssl-dev libffi-dev python3-dev

CentOS, RedHat:

sudo yum install gcc libffi-devel python-devel openssl-devel

3. ImportError: No module named 'cryptography'

This is a component that lacks encryption. You can use pip to install it.

sudo pip3 install cryptography

4. ImportError: No module named 'packaging'

The packaging package is missing. It provides the core functions of the Python package and can be installed using pip.

sudo pip3 install packaging

5. ImportError: No module named 'appdirs'

The appdirs package is missing. It is used to determine the file directory. You can use pip to install the package separately.

sudo pip3 install appdirs

The above is all the content of this article. I hope it will be helpful for your learning and support for helping customers.