Scrapy crawler -- 02

The most basic part of a crawler is to download the web page, and the most important part is to filter-get the information we need.

Scrapy provides the following functions:

First, we need to define items:

ItemsAre containers that will be loaded with the scraped data; they work like simple Python dicts but provide additional protection against populating undeclared fields, to prevent typos.

From the official website, it means that iteams is used to store the captured data structure and provides additional type protection relative to the python dictionary type. (This specific protection method is to be studied)

Example:

Project/items. py

import scrapyclass DmozItem(scrapy.Item):    title = scrapy.Field()    link = scrapy.Field()    desc = scrapy.Field()

Then we need to write a spider, capture the webpage, select information, and put it into items.

Example:

import scrapyclass DmozSpider(scrapy.Spider):    name = "dmoz"    allowed_domains = ["dmoz.org"]    start_urls = [        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"    ]    def parse(self, response):        filename = response.url.split("/")[-2]        with open(filename, ‘wb‘) as f:            f.write(response.body)

Note:

1. Name spider name, which must be unique in this project. The reason will be known later.

2. Allowed_domains. This is the domain name setting, that is, whether to capture other domain names. Generally, you can create a domain name to be the address in start_urls.

3. Start_urls: this is a list that indicates the starting position of the scrapy web page to capture. Take the following example.
4. In my opinion, the parse function is similar to the interface concept in Java. scrapy sends a request to the website, and then calls the parse function in the dsf-spider to process the returned response.

Start the spider:

scrapy crawl dmoz

The normal output is as follows:

2014-01-23 18:13:07-0400 [scrapy] INFO: Scrapy started (bot: tutorial)2014-01-23 18:13:07-0400 [scrapy] INFO: Optional features available: ...2014-01-23 18:13:07-0400 [scrapy] INFO: Overridden settings: {}2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled extensions: ...2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled downloader middlewares: ...2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled spider middlewares: ...2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled item pipelines: ...2014-01-23 18:13:07-0400 [dmoz] INFO: Spider opened2014-01-23 18:13:08-0400 [dmoz] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/> (referer: None)2014-01-23 18:13:09-0400 [dmoz] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> (referer: None)2014-01-23 18:13:09-0400 [dmoz] INFO: Closing spider (finished)

To be continued

Scrapy crawler -- 02