"Reprint" Python3 installation scrapy windows32 bit crawl Pit

Python3 installation of Scrapy windows32-bit crawl pitsoriginal November 06, 2016 01:38:08

• Label:
• Scrapy/
• Windows/
• Python/
• Open Source Framework/
• Web crawler

Long heard that Scrapy does not support Python3, and Scrapy as an excellent open source framework, not on the new Python integration is a pity. Recently just want to start learning web crawler, online and so many tutorials, so today to crawl this pit.
First of all, of course, install Python, the official website can be installed directly the latest 3.5.2. After the installation is complete, you can execute the command in CMD after the python succeeds

install scrapy

• 1

OK, then wait for the results to be executed. Young man, do you think this is over, too too navie. Wait more than 10 minutes, look back, error,

Word, what's up with the swelling? Twisted Download 98%,lxml Newspaper exception, what the heck is this?
Don't worry, look for degrees Niang, originally these are run scrapy rely on the library. Okay, here's the text, start climbing the pit ~
First degree Niang, Scrapy relies on the library has twisted, lxml, and twisted relies on the library has zope.interface, Pyopenssl. Everyone crossing is not already slightly wearies people, do not hurry, the future is a good drop, but the course of learning is always tortuous ~

First, install the Twisted Library

1. Install the zope.interface.
We can first execute the command in the Python environment import zope.interface to determine if Zope.interface has been installed, and if not, then congratulate you on your skip this step and go directly to the next step. If there is an error and there is a problem, then we will solve the hole.

Download Zope.interface here from Zope.interface. OK, when you open the link click on the top right corner of the light green button downloads, and dumbfounded!

Because the files in the Zope download list are obviously only supported to py3.4!!! What to do? It is only possible to download the file first and then see if it is good for py3.5 (Nieyibahan!). ）。 Here we download the last file, and then double-click Execute.

OK, and Navie again, so this method side abandon.
Next, try installing the Zope.interface WHL file directly. The premise requires your Python installation wheel, if not installed wheel can execute commands in cmd pip install wheel to install. We download ... (10,000 words omitted here)

Nani, Zope WHL file actually has support py3.5, but EXE file did not, simply ... Toss half a day to waste food. OK, let's download the selected WHL file directly and execute the following command in CMD

pip install D:\Study\spider\scrapy\zope.interface-4.3.2-cp35-cp35m-win32.whl

• 1

See successfully word, OK, zope.interface finally fix.

2. Install the PYOPENSSL.

Here is still the first download Pyopenssl, open the link, still click on the upper right corner of the light green button downloads, and then download the first file in the list.

After the download is complete, then the same ritual opens cmd execution command

pip install D:\Study\spider\scrapy\pyOpenSSL-16.2.0-py2.py3-none-any.whl

• 1

Then wait for the successful installation to complete, OK, but often backfired ~
。
Baa Riga? Helpless again to help degrees Niang. (All the shining of the crowd do not need to find me, I am in the mother's warm arms)
Suspect that the cryptography library is not installed, so execute the command pip install cryptography

Installation cryptography is successful and you can see that its version is 1.5.2. Then went back to install Pyopenssl, execute the command

pip install D:\Study\spider\scrapy\pyOpenSSL-16.2.0-py2.py3-none-any.whl

• 1

Well, happiness comes too suddenly and is caught off guard. Sure enough, still need to meditate more thinking. Next
3. Install the twisted.
Continue to the official website to download twisted installation files.

The py3.5 version of Twisted is not found here, so download the latest twisted version 16.5 directly.
We directly download the first item of the WHL file in the list, and after the download is complete, execute the command in CMD:

pip install D:\Study\spider\scrapy\Twisted-16.5.0-cp27-cp27m-win_amd64.whl

• 1

Small stool in front, waiting for the result ...

Version not supported, looked at the installation of the twisted is 64-bit. So go on, look for the 32-bit twisted.
Emperor is not a conscientious person, found here in the 32-bit twisted. Then decisive download down, execute the command installation.


Well, it seems because the version of Python installed is 3.5, and the version of twisted does not match. However, the latest version of the official website 16.5 means that the financial support to the python2.7 version, but also 64-bit.
Ma Ma, you hold away, I will cry to cry will 、、、
After several hours of tossing, finally found the wisdom of the People Power is infinite, found an article, which provides a wild version of the twisted cp35 available for download.

Well, please allow me to show off my dewford knowledge: The true son is pale no place to find, have to take all the time ~
OK, after the download is complete, execute the command:

pip install D:\Study\spider\scrapy\Twisted-16.5.0-cp35-cp35m-win32.whl

• 1

Long-lost successfully, too miss you Ah!

Second, installation lxml

First download the lxml installation file,

Execute command after download is complete

pip install D:\Study\spider\scrapy\lxml-3.6.4-cp35-cp35m-win32.whl

• 1

OK, honestly, it's getting more and more rare. Pip this feature ~

Third, the final installation of Scrapy

The Great Wall finally came to the last step, so long bang, to download scrapy.

After the download is complete, we execute the command in cmd

pip install D:\Study\spider\scrapy\Scrapy-1.2.1-py2.py3-none-any.whl

• 1

If the installation process errors, do not worry, that is because of network reasons, the download timed out, at this time to try a few more orders, anyway cattle days to learn the Scriptures have finally got a difficult.

Now, all the dependencies were successfully installed, including Scapy. The red part of the above means that there is no successful compilation of the Pydispatcher wheel file, but finally Python automatically reinstalled the Pydispatcher, and there are successfully words.

So, in the end, the last, and so much more, is to test whether the scrapy is working properly. Enter scrapy directly in CMD and see as shown:

Well, congratulations, you're done!

"Reprint" Python3 installation scrapy windows32 bit crawl Pit