Introduction to scrapy1.1 and scrapy1.1

Introduction to scrapy1.1 and scrapy1.1

Today, we have successfully installed scrapy and tested it. It takes a long time to complete the verification.

In fact, the best teacher is scrapy's help document. As long as you read and follow the document, it will take a while!

Help Document Download see http://download.csdn.net/detail/flyinghorse_2012/9566467

 

0. Create a new folder to store related files and name it test

 

1. Build a scrapy project

Run the following command:

Scrapy startproject tutorial

The effect is as follows:

 

2. Build a spider

Run the following command:

Scrapy genspider dmoz dw..org

Format requirements: scrapy genspider spidername spiderwebsite

Spidername must be unique, and spiderwebsite can be set at will, corresponding to allowed_domains in dmoz. py.

The effect is as follows:

 

3. modify items. py

Find... test \ tutorial \ items. py and modify the file content:

Import scrapy

Class TutorialItem (scrapy. Item ):
Title = scrapy. Field ()
Link = scrapy. Field ()
Desc = scrapy. Field ()

Save.

 

4. Modify dmoz. py

Find... \ test \ tutorial \ spiders \ dmoz. py and modify the file content:

#-*-Coding: UTF-8 -*-
Import scrapy

Class DmozSpider (scrapy. Spider ):
Name = "dmoz"
Allowed_domains = ["dw..org"]
Start_urls = (
"Http://www.dmoz.org/Computers/Programming/Languages/Python/Books ",
Http://www.dmoz.org/Computers/Programming/Languages/Python/Resources"
)

Def parse (self, response ):
Filename = response. url. split ("/") [-2] + '.html'
With open (filename, 'wb ') as f:
F. write (response. body)

Save.

 

5. Run Crawler

Scrapy crawl dmoz

Format requirements: scrapy crawl spidername

Spidername is the spidername in step 2.

The effect is as follows:

Two html files have been generated successfully, and the webpage content has been crawled.