Run the first scrapy project under the pro-Test--pycharm ©seven_clear

Recently in learning Scrapy, just want to use Pycharm debugging, but do not know how to do, from the online search a lot of methods, here summed up a I try to succeed.

First of all of course installs Scrapy, installs the tutorial what online a lot of, here recommend a detailed: http://blog.csdn.net/php_fly/article/details/19364913, inside has the resources of Ann's thing. There are a few points to note: 1) Python Configuration environment variables it is best to add the Scripts path, as mine is D:\Python27\Scripts. Because the installation of scrapy to use, 2) Zope.interface-4.0.5-py2.7-win32.egg when the open mode selected scripts path Easy_install.exe. The rest should be fine.

The general process is: Create a new Scrapy project with the command line, and write your own crawler code, with Pycharm open the project.

Find the directory where you want to build the project, and then start cmd in that directory, enter scrapy startproject name (custom project name), no surprise you will see the name folder in your directory, this is your scrapy project, open the folder, There will be a folder with the same name and a scrapy.cfg file. ©seven_clear

Then open the project with Pycharm, create your own crawler, for reference: http://scrapy-chs.readthedocs.io/zh_CN/latest/intro/ Tutorial.html, but I'm a 0.20.2 version, the code is slightly different when defining your own reptiles:

#Coding:utf-8 fromScrapy.spiderImportBasespiderclassDmozspider (basespider): Name="DMOZ"Allowed_domains= ["dmoz.org"] Start_urls= [        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"    ]    defParse (self, Response): filename= Response.url.split ("/") [-2] Open (filename,'WB'). Write (Response.body)

The import is Basespider, and the rest is the code on the document. When editing items.py, write your own item class under the original class:

#Define Here the models for your scraped items##See documentation in:#http://doc.scrapy.org/en/latest/topics/items.html fromScrapy.itemImportItem, FieldclassTestscrapyitem (Item):#Define the fields for your item here is like:    #name = Field ()    PassclassDmozitem (Item): Title=Field () Link=Field () desc= Field ()

OK, now it's running, just write a command line, for convenience, write into a python file, name customization (can be named Main), and scrapy.cfg in the sibling directory, the content is two lines:

# Coding:utf-8  from Import Cmdlinecmdline.execute ("scrapy crawl dmoz". Split ())

DMOZ is the name of a reptile, variable. This will allow us to run our main.py. How, simple, some things have to be done by themselves, back to find it is really simple. ©seven_clear

Run the first scrapy project under the pro-Test--pycharm ©seven_clear