How the Scrapy framework installs the configuration in Python

In the Python learning Group found a lot of learning web crawler technology of children's boots are not understand the Python crawler framework scrapy installation configuration, in the early stages of learning Python crawler, we use Urllib and URLLIB2 library and regular expression can be completed, But encounter more powerful crawler tool-crawler frame scrapy, this installation process is also painstakingly, in this collation as follows.

Windows platform:

My system is Win7, first of all, you have to have Python, I use 2.7.7 version, Python3 similar, just some source files are different.

Official documents: http://doc.scrapy.org/en/latest/intro/install.html, the most authoritative, the following is my personal experience process.

1. Install Python

Installation process I don't have to say much, I have already installed Python 2.7.7 version, after installation, remember to configure the environment variables, such as my installation on the D Drive, D:\python2.7.7, the following two paths are added to the path variable

D:\python2.7.7;d:\python2.7.7\scripts

Once configured, enter python–version on the command line and if there is no error, the installation succeeds

2. Installing Pywin32

Under Windows, you must install PYWIN32, install address: http://sourceforge.net/projects/pywin32/

Download the corresponding version of the Pywin32, directly double-click the installation can be completed after installation verification:

Under the Python command line, enter

Import win32com

If no error is indicated, the installation is successful

3. Install Pip

PIP is the tool used to install other necessary packages, first download get-pip.py

After downloading, select the path of the file and execute the following command

Python get-pip.py

The PIP is installed after the command is executed, and at the same time it helps you install the Setuptools after the installation is finished and then executes PIP--version on the command line

If prompted as follows, the installation is successful, if the prompt is not internal or external commands, then check the environment variables are not configured, there are two paths.

4. Installing Pyopenssl

Under Windows, there is no preinstalled Pyopenssl, and Linux is already installed.

Installation Address: Https://launchpad.net/pyopenssl

5. Installing lxml

Lxml's detailed introduction to me is a library written in Python that can quickly and flexibly process XML directly by executing the following commands

Pip Install lxml

To complete the installation, if you are prompted that the Microsoft Visual C + + library is not installed, click I download the supported libraries.

6. Installing Scrapy

Finally is the exciting moment, the top of the cushion is done, we can finally enjoy the fruit of victory!

Execute the following command

Pip Install Scrapy

Pip will download additional dependent packages, these will not be installed manually, wait for a while, we are done!

7. Verifying the Installation

Input Scrapy

If you are prompted with the following command, it proves that the installation was successful and if it fails, please check the above steps for any omissions.

Linux Ubuntu Platform:

Linux installation is very simple, only a few commands to execute several

1. Install Python

sudo apt-get install python2.7 Python2.7-dev

2. Install Pip

First download get-pip.py

After downloading, select the path of the file and execute the following command

sudo python get-pip.py

3. Install scrapy directly

Because lxml and OpenSSL are already preloaded under Linux

If you want to verify lxml, you can enter each

sudo pip install lxml

The following prompt appears to prove that the installation was successful

Requirement already satisfied (use--upgrade to upgrade): Lxml in/usr/lib/python2.7/dist-packages

If you want to verify OpenSSL, enter OpenSSL directly, and if you jump to the OpenSSL command line, the installation succeeds.

Next, you can install scrapy directly

sudo pip install Scrapy

After the installation is complete, enter scrapy

Note that here Linux does not enter Scrapy,linux is still strictly case-sensitive, thanks to Kamen Children's shoes reminders.

If the following prompt appears, this proves that the installation was successful

Usage:

scrapy [Options] [args]

Available commands:

Bench Run Quick Benchmark test

Fetch fetch a URL using the Scrapy downloader

Runspider Run a self-contained spider (without creating a project)

Settings Get Settings values

Shell Interactive Scraping Console

Startproject Create New Project

Version Print scrapy version

View Open URL in Browser, as seen by Scrapy

[More] More commands available if run from project directory

If you have any questions, please leave a message! I wish you a smooth installation of small partners!

How the Scrapy framework installs the configuration in Python