Python使用scrapy抓取網站sitemap資訊的方法

本文執行個體講述了Python使用scrapy抓取網站sitemap資訊的方法。分享給大家供大家參考。具體如下：

import refrom scrapy.spider import BaseSpiderfrom scrapy import logfrom scrapy.utils.response import body_or_strfrom scrapy.http import Requestfrom scrapy.selector import HtmlXPathSelectorclass SitemapSpider(BaseSpider): name = "SitemapSpider" start_urls = ["http://www.domain.com/sitemap.xml"] def parse(self, response):  nodename = 'loc'  text = body_or_str(response)  r = re.compile(r"(<%s[\s>])(.*?)()"%(nodename,nodename),re.DOTALL)  for match in r.finditer(text):   url = match.group(2)   yield Request(url, callback=self.parse_page) def parse_page(self, response):    hxs = HtmlXPathSelector(response)    #Mock Item  blah = Item()  #Do all your page parsing and selecting the elemtents you want    blash.divText = hxs.select('//div/text()').extract()[0]  yield blah

希望本文所述對大家的Python程式設計有所協助。

