Python crawler advanced two Pyspider framework installation configuration

About

First, attach the address of the project and the official documentation

Pyspider Official documents

Install 1. Pip

First make sure you have the PIP installed, and if it is not installed, refer to

PIP installation

2. Phantomjs

PHANTOMJS is a WebKit-based server-side JavaScript API. It fully supports the web without browser support, and its fast, native support for a variety of Web standards: DOM processing, CSS selectors, JSON, Canvas, and SVG. PHANTOMJS can be used for page automation, network monitoring, web screen screenshots, and no interface testing.

Installation

With the official installation, if you are an Ubuntu or Mac OS x user, you can install it directly with the command

Ubuntu:

1	sudo apt-get install phantomjs

Mac OS X:

1	Brew Install phantomjs

3. Pyspider

Directly with PIP installation

1	Pip install pyspider

If you are an Ubuntu user, please install the following Support class library in advance.

1	sudo apt-get install python python-< Span class= "crayon-e" >dev python-distribute python-pip libcurl4 -openssl-dev libxml2-dev libxslt1-dev python- lxml

Test

If the installation process does not prompt any errors, then prove some OK.

Command Line Input

1	Pyspider All

Then the browser accesses http://localhost:5000

Look at the effect, if you can normally appear Pyspider page, that proves everything OK

Here is a picture, this is I wrote a few crawlers after the interface.

OK, next I'll cover the use of this framework.

Common errors

One of the mistakes I've ever encountered:

Pyspider HTTP 599:ssl certificate problem wrong solution, later in the author that sent issue got the answer, the other temporarily no problem.

However, I found that some of the small partners raised a variety of problems ah, but I did not meet, I again win10,linux ubuntu,linux centos,mac OS x run successfully. But there were some strange problems, running and running and collapsing, and I was wondering about it.

If you have a problem, you can look at the author of the project there is no similar issue, but also recommend that you go directly to the author of GitHub on the hair issue.

After all, this framework was not written by me.

Attach issue address here:

Pyspider Issue

Reprint: Quiet Find? Python crawler advanced two Pyspider framework installation configuration

Python crawler advanced two Pyspider framework installation configuration