Beginner crawlers We can use the Urllib and URLLIB2 libraries and regular expressions to complete, but there are more powerful tools, crawler framework Scrapy, this installation process is also painstakingly, in this collation as follows.
Windows Platform:
My system is Win7, first of all, you have to have Python, I use 2.7.7 version, Python3 similar, just some source files are different.
Official documents: http://doc.scrapy.org/en/latest/intro/install.html, the most authoritative, the following is my personal experience process.
1. Install Python
Installation process I don't have to say much, I have already installed Python 2.7.7 version, after installation, remember to configure the environment variables, such as my installation on the D Drive, D:\python2.7.7, the following two paths are added to the path variable
? D:\python2. 7.7;D: \python2. 7.7\scripts
Once configured, enter python–version on the command line and if there is no error, the installation succeeds
2. Installing Pywin32
Under Windows, you must install PYWIN32, install address: http://sourceforge.net/projects/pywin32/
Download the corresponding version of the Pywin32, directly double-click the installation can be completed after installation verification:
Under the Python command line, enter
Import win32com
If no error is indicated, the installation is successful
3. Install Pip
PIP is the tool used to install other necessary packages, first download get-pip.py
After downloading, select the path of the file and execute the following command
Get-pip.py
The PIP is installed after the command is executed, and at the same time it installs the Setuptools for you.
Executes on the command line after the installation is finished
? PIP--version
If prompted as follows, the installation is successful, if the prompt is not internal or external commands, then check the environment variables are not configured, there are two paths.
4. Installing Pyopenssl
Under Windows, there is no preinstalled Pyopenssl, and Linux is already installed.
Installation Address: Https://launchpad.net/pyopenssl
5. Installing lxml
Lxml's detailed introduction to me, is a library written in Python that allows you to quickly and flexibly process XML
Execute the following command directly
Pip Install lxml
To complete the installation, if you are prompted that the Microsoft Visual C + + library is not installed, click I download the supported libraries.
6. Installing Scrapy
Finally is the exciting moment, the top of the cushion is done, we can finally enjoy the fruit of victory!
Execute the following command
? pip Install Scrapy
Pip will download additional dependent packages, these will not be installed manually, wait for a while, we are done!
7. Verifying the Installation
Input Scrapy
If you are prompted with the following command, it proves that the installation was successful and if it fails, please check the above steps for any omissions.
Linux Ubuntu Platform:
Linux installation is very simple, only a few commands to execute several
1. Install Python
? sudo apt-get install Python2. 7 Python2. 7-dev
2. Install Pip
First download get-pip.py
After downloading, select the path of the file and execute the following command
Get-pip.py
3. Install scrapy directly
Because lxml and OPENSSL are already preloaded under Linux
If you want to verify lxml, you can enter each
? sudo pip install lxml
The following prompt appears to prove that the installation was successful
in/usr/lib/python2. 7/dist-packages
If you want to verify OpenSSL, enter OpenSSL directly, and if you jump to the OpenSSL command line, the installation succeeds.
Next, you can install scrapy directly
? sudo pip install Scrapy?
After the installation is complete, enter scrapy
Note that here Linux does not enter Scrapy,linux is still strictly case-sensitive, thanks to Kamen Children's shoes reminders.
If the following prompt appears, this proves that the installation was successful
<command>using the scrapy Downloader runspider Run aself-new in as seen by Scrapy from project directory?
As follows
If you have any questions, please leave a message! I wish you a smooth installation of small partners!
Sesame Http:python Reptile Advanced scrapy Frame installation configuration