Python3 under Scrapy crawler (first volume: Installation issues)

General reptiles are used Urllib package, requests package with regular. BeautifulSoup and other packages mixed use, to achieve the reptile effect, but there is a framework who still use the original Ah, now we talk about Scrapy framework crawler,

Now Python3 compatibility comes up, scrapy not only support Python2 version, have new not old, now say let a lot of people placing installation problem, many people start to install not understand,

The version currently in use is PYTHON3.5 and is installed with PIP3

Installation steps:

1 Installing Wheel

PIP3 Install Wheel

2 Installing Twisted

PIP3 Install TWISTED-17.9.0-CP35-CP35M-WIN_AMD64.WHL

3 Installing lxml

PIP3 Install LXML-4.1.1-CP35-CP35M-WIN_AMD64.WHL

4 Installing Scrapy

PIP3 Install Scrapy

This allows you to successfully install the scrapy, you can create files, but you are not allowed to crawl, once the execution of the crawler file will be error

5 Installing Pywin32

Just the next step.

Then we create the Scrapy folder

Go to the python3.5 file path, if your computer is installed at the same time 2, 3 version must pay attention to the problem. Two version of the environment conflict issues, once the Python3 version of the Scrapy run under the Python2 version incompatibility problem, there will be no moudle error

The path switches to the PYTHON3 running environment:

Scrapy startproject filename

Terminal into the filename directory

Scrapy genspider-t Basic CRAWL1 webname.com

A crawler script file will be created

I'm not going to introduce a few files in the folder.

I say the application of basic reptile setting.py

You can now crawl the page normally.

Python3 under Scrapy crawler (first volume: Installation issues)