Python scrapy crawls dynamic pages

Python scrapy crawls dynamic pages

Preface: in addition to my recent work, a friend of the opposite sex needs to crawl dynamic web pages and enter keywords to crawl some patent descriptions of a patented website under this keyword. In the past, python urllib2 was broken, but it was only broken for static Web pages. However, for dynamic web pages generated using js and other products, it seems that it is not good (I have never tried it ). Then I found some information on the Internet, and found that scrapy combined with selenium package seems to be OK. (The reason for this is that the halogen master has not been implemented yet, so we should record it first .)

#==========================================According to the simple introduction on the official website for personal understanding ============== ====================

First, install two scrapy and selenium packages:

If anaconda, pip, and easy_intasll have been installed in ubuntu, install anaconda in one step (or easy_install ):

pip install -U seleniumpip install Scrapyeasy_install  -U seleniumeasy_install  Scrapy

Next, use scrapy to create a project, and run the following command on the terminal to create a project:

scrapy startproject tutorial

The following folders are automatically generated:

Figure 1: folder after the project is created

Again, start to write the project:

Some variables need to be defined in the items. py file:

import scrapyclass DmozItem(scrapy.Item):    title = scrapy.Field()    link = scrapy.Field()    desc = scrapy.Field()

In the folder tutorial/spidersCreate dmoz_spider.pyFile:

 

import scrapyclass DmozSpider(scrapy.Spider):    name = "dmoz"    allowed_domains = ["dmoz.org"]    start_urls = [        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"    ]    def parse(self, response):        filename = response.url.split("/")[-2]        with open(filename, 'wb') as f:            f.write(response.body)

In this file, you need to define three variables: name, start_urls, and parse.

Finally, run the terminal in the outermost Folder:

scrapy crawl dmoz

Dmoz is a folder tutorial/spidersThe value of name, one of the important variables in the DmozSpider class in the newly created file.

You can start crawling.

#===================================================== ======

Boyou: http://chenqx.github.io/2014/12/23/Spider-Advanced-for-Dynamic-Website-Crawling/

This section describes how to capture dynamic website content and shares the complete project code on github. (Download the code first and read the document from the code)

Its gouwu.sogou.com task dynamically captures page information.

The halogen master changes its tasks.

The analysis shows that Boyou's Code does not find the code where the crawled dynamic page information exists,

Add your own code to the spider. py file:

    def parse(self, response):        #crawl all display page        for link in self.link_extractor['page_down'].extract_links(response):            yield Request(url = link.url, callback=self.parse)        #browser        self.browser.get(response.url)        time.sleep(5)        # get the data and write it to scrapy items        etaoItem_loader = ItemLoader(item=EtaoItem(), response = response)        url = str(response.url)        etaoItem_loader.add_value('url', url)        etaoItem_loader.add_xpath('title', self._x_query['title'])        etaoItem_loader.add_xpath('name', self._x_query['name'])        etaoItem_loader.add_xpath('price', self._x_query['price'])        #====================================#        for link in self.link_extractor['page_down'].extract_links(response):#            yield Request(url = link.url, callback = self.parse_detail)        for sel in response.xpath('//ul/li'):            title = sel.xpath('a/text()').extract()            link2 = sel.xpath('a/@href').extract()            desc = sel.xpath('text()').extract()            for i in title:                print i,            for j in link2:                print j,"+++++++++++++"        #====================================        yield etaoItem_loader.load_item()

You can analyze some items, but it is not enough. You still need to analyze the source code of the returned dynamic page, change the extractor and selector (not started yet) to get the desired result. The selenium package seems useless.

#===================================================== ======

Halogen master references:

Scrapy Website: http://doc.scrapy.org/en/latest/intro/tutorial.html

Selenium Official Website: http://selenium-python.readthedocs.org/

Scrapy selector: http://doc.scrapy.org/en/0.24/topics/selectors.html#topics-selectors

Blog: http://chenqx.github.io/2014/12/23/Spider-Advanced-for-Dynamic-Website-Crawling/
Blog: http://chenqx.github.io/2014/11/09/Scrapy-Tutorial-for-BBSSpider/

Selenium: http://www.cnblogs.com/fnng/archive/2013/05/29/3106515.html

Access problems: http://my.oschina.net/HappyRoad/blog/173510