Crawler-Scrapy framework and scrapy framework

Crawler-Scrapy framework and scrapy framework

• Scrapy is an application framework written in Python to crawl website data and extract structural data. It is widely used.
• With the strength of the Framework, users can easily implement a crawler by customizing and developing several modules to crawl webpage content and various images, which is very convenient.
• Scrapy uses the Twisted Asynchronous Network Framework to process network communication, which can accelerate the download speed. It does not need to implement the asynchronous framework by itself, and contains various middleware interfaces, which can flexibly meet various requirements.

Scrapy Architecture

• Scrapy Engine: responsible for communications, signal, data transmission among Spider, ItemPipeline, Downloader, and Scheduler.
• Schedest.
• Downloader: downloads all Resquests requests sent by the Scrapy Engine, returns the obtained Responses to the Scrapy Engine, and the Engine sends the Downloader to the Spider for processing.
• Spider (crawler): it processes all Responses, analyzes and extracts data from it, obtains the data required by the Item field, and submits the URL to the engine, go to sched again ).
• Item Pipeline: It processes the items obtained by the Spider and carries out post-processing (detailed analysis, filtering, storage, etc.
• Downloader Middlewares (download middleware): You can use it as a component that can customize the download Extension function.
• Spider Middlewares (Spider middleware): You can understand it as a function component that allows custom extension and operation engine to communicate with Spider (for example, Responses for entering Spider and Resquests for going out from Spider ).

Four steps are required to create a Scrapy crawler:

• Create a project (scrapy startproject project name): Create a New crawler Project
• Define goals (compile items. py): define the goals you want to crawl
• Create a crawler (spiders/xxxspider. py): Make a crawler and start crawling the webpage.
• Storage content (pipelines. py): Design pipelines to store crawled content

Entry case

Objectives:

• Create a Scrapy Project
• Define extracted structured data (Item)
• Write a website crawler and extract structured data (Item)
• Write Item Pipelines to store extracted items (structured data)

1. Create a project (scrapy startproject)

A new Scrapy project must be created before crawling. Go to the custom project directory and run the following command:

scrapy startrpoject mySpider

MySpider is the project name. You can see that a mySpider folder is created. The directory structure is roughly as follows:

Download to briefly introduce the functions of each major file:

Scrapy. cfg: project configuration file mySpider/: Python module of the project. The Code mySpider/items will be referenced here. py: the project's target file mySpider/middlewares. py: the project's MPs queue file mySpider/pipelines. py: the project's MPs queue file mySpider/settings. py: Project setting file mySpider/spiders/: stores the crawler code directory

2. Define the target (mySpider/items. py)

Here we take the Capture: All the lecturer's name, title and personal information in the http://www.itcast.cn/channel/teacher.shtml website as an example.

1. open items. py in the mySpider directory

2. Item defines structured data fields to store crawled data, which is a bit like dict in Python, but provides some additional protection to reduce errors.

3. You can create a scrapy. Item class and define the class attribute of scrapy. Field to define an Item.

4. Next, create an ItcastItem class and a model ).

Import scrapyclass ItcastItem (scrapy. item): # define the fields for your item here like: # instructor name = scrapy. field () # title level = scrapy. field () # Introduction info = scrapy. field ()

3. Create a crawler (spiders/itcastSpider. py)

The crawler function can be divided into two steps:

1. Data crawling

Enter a command in the current directory. A crawler named itcast will be created in the mySpider/spider directory and the crawling range of the domain will be specified:

scrapy genspider itcast "itcast.cn"

Open itcast. py in the mySpider/spider directory. The following code is added by default:

import scrapyclass ItcastSpider(scrapy.Spider):    name = "itcast"    allowed_domains = ["itcast.cn"]    start_urls = (        'http://www.itcast.cn/',    )    def parse(self, response):        pass

You can also create itcast. py and write the above Code on your own, but using commands can save the trouble of writing fixed code on the cloud.

To create a Spider, you must use the scrapy. Spider class to create a subclass and determine three mandatory attributes and a method.

Name = "": The identification name of this crawler, which must be unique, must be defined for different crawlers. Allow_domains = []: indicates the domain name range of the search, that is, the restricted area of the crawler. It requires that the crawler only crawls the webpage under the domain name, And the nonexistent URL will be ignored. Start_urls = (): the URL tuples/list crawled. Crawlers start to capture data from here, so the data downloaded for the first time will start from these urls. Other sub-URLs are generated from these starting URLs. Parse (self, response): Resolution method. Each initial URL is called after download. When called, the Response object returned from each URL is passed as a unique parameter, the main functions are as follows: 1. parses the returned webpage data (response. body), extract structured data (generate item) 2. generate a URL request for the next page.

Change the value of start_urls to the first url to be crawled.

Start_urls = ("http://www.itcast.cn/channel/teacher/shtml",) or start_urls = ["http://www.itcast.cn/channel/teacher/shtml"]

2. Fetch data

After crawling the entire webpage, the next step is the process. First, observe the page source code:

<div class="li_txt">    
The code of the parse function is as follows:
    def parse(self, response):        item = ItcastspiderItem()        for each in response.xpath("//div[@class='li_txt']"):            name = each.xpath("h3/text()").extract()            level = each.xpath("h4/text()").extract()            info = each.xpath("p/text()").extract()            item["name"] = name[0].strip()            item["level"] = level[0].strip()            item["info"] = info[0].strip()            yield item
Item Pipeline
After an Item is collected in Spider, it will be passed to Item Pipeline, which processes items in the defined order.
Each Item Pipeline is a Python class that implements a simple method. For example, it is determined that this Item is discarded and stored. The following are some typical application of item pipeline:
Verify the crawled data (check that the item contains certain fields, such as the name field)
Re-Query
Save the crawling result to a file or database.
It is very easy to compile the item pipeline. The item pipeline component is an independent Python class. The process_item () method must be implemented:
#-*-Coding: UTF-8-*-# Define your item pipelines here # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: http://doc.scrapy.org/en/latest/topics/item-pipeline.htmlimport jsonclass ItcastspiderPipeline (object ): def _ init _ (self): self. file = open ("Chuanzhi instructor. json "," w ", encoding =" UTF-8 ") self. first_flag = True def process_item (self, item, spider): if self. first_flag: self. first_flag = False content = "[\ n" + json. dumps (dict (item), ensure_ascii = False) else: content = ", \ n" + json. dumps (dict (item), ensure_ascii = False) self. file. write (content) return item def close_spider (self, spider): self. file. write ("\ n]") self. file. close ()
Enable an Item Pipeline component
To enable the Item Pipeline component, you must add its class to the ITEM_PIPELINES configuration in the settings. py file, as shown in the following example:
# Configure item pipelines# See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.htmlITEM_PIPELINES = {    'itcastSpider.pipelines.ItcastspiderPipeline': 300,}
The integer values of each class in the allocation group determine the order in which they run. items are defined in the ascending order of numbers. Through pipeline, these numbers are usually defined within the range of 0 to (0 to are set at will, the lower the value, the higher the component priority)
Start Crawler
scrapy crawl teacher
Check whether the local disk has a generated intelligence instructor. json