Install Scrapy in Ubuntu

I have been learning about crawlers recently. I have heard that Python write crawlers are extremely cool (it seems that python is both cool, but it is also true that python class libraries are very rich and do not need to duplicate the wheel ), there is also a powerful Scrapy framework, so I decided to try it. To use Scrapy, the first thing is to install Scrapy. I tried to install Scrapy on Windows and Ubuntu. This article first describes how to install Ubuntu, which is much easier than installing Windows. It will also explain in detail how to find

I have heard of crawlers recently.PythonThe write crawler is superb (it seems that python is good, but it is also true that python class libraries are rich and don't need to be repeated). There is also a powerful frameworkScrapySo I decided to try it.

To useScrapyThe first thing, of course, is to installScrapy, TriedWindowsAndUbuntuThis article first introducesUbuntuThanWindowsInstallation is too simple. It will also explain in detail howWindows.

Official introduction, installed inScrapyInstall a series of dependencies.
*Python 2.7:ScrapyYesPythonThe framework must be installed first.PythonBut becauseScrapyCurrently only supportedPython2.7First, make sure that you installPython 2.7
*lxml: MajorityLinuxThe release comeslxml
*OpenSSL: Apart fromwindowsOther systems have provided
*Python Package: Pip and setuptools.pipDependencysetuptools, So installpipAutomatically installedsetuptools

As shown in the preceding dependency, installing Scrapy in a non-windows environment is relatively simple.pipYou can.ScrapyUsepipComplete the installation.

Check ScrapyInstall dependency?

You may not worry about whether your computer has been installed or not, and the dependency mentioned above already exists. You can use the following method to check whether this article usesUbuntu 14.04.

Check PythonVersion

$ python --version

If the following output is displayedPythonThe environment has been installed. What I show here isPython 2.7.6The version is also2.7. If the following information is not displayed, ask the reader to install it on BaiduPython, This article does not introducePythonInstallation (a bunch of online searches ).

Check lxmlAnd OpenSSLInstall or not

Assume that you have installedPython, Enterpython, EnterPython.

Enterimport lxmlAndimport OpenSSLIf no error is reported, both Dependencies are installed.

Install python-devAnd libevent

python-devYeslinuxDevelopment onpythonImportant tools. You need to install
* You need to install a python class library outside the source, which contains the c/c ++ file that needs to be compiled to call the python api.
* A program compiled by yourself needs to be linked to libpythonXX. (a | so)

libeventIt is a high-performance network library starting from time, and is used at the underlying layer of many frameworks.libevent

The above two libraries need to be installed, or an error will be reported later. Use the following command to install

$sudo apt-get install python-dev$sudo apt-get install libevent-dev

Install pip

BecauseScrapyAvailablepipTherefore, we need to installpip, You can use the following command to installpip

$ sudo apt-get install python-pip

Use pipInstall Scrapy

Use the following command to installScrapy.

$ sudo pip install scrapy

Remember to getrootPermission. Otherwise, the following error occurs.

So farscrapyThe installation is complete. Run the following command to checkScrapyWhether the installation is successful.

$ scrapy version

The following result indicates that the installation is successful. The installation version here is1.02

CentOS 6.4 installation setup Scrapy 0.22 environment http://www.linuxidc.com/Linux/2015-08/120897.htm

How to install Web Crawler in Ubuntu 14.04 LTS: Scrapy http://www.linuxidc.com/Linux/2015-03/115306.htm

For more information about Ubuntu, see Ubuntu special page http://www.linuxidc.com/topicnews.aspx? Tid = 2

This article permanently updates the link address: Http://www.linuxidc.com/Linux/2015-08/122442.htm