Python scrapy Crawler: Demo for synchronous and asynchronous paging

The paging interaction is synchronous and asynchronous when the data is requested, while the whole page refreshes when synchronizing, and the page is refreshed asynchronously. For these two paging data in the crawler, the manner of processing is not the same. Demo only for learning, domain name all to test

Synchronizing paging

When paging is synchronized, the whole page refreshes and the URL address bar changes

The data object that the crawler parses is HTML

Test scenario: Crawl a Job site in the Beijing area of Java jobs

#coding =utf-8import scrapyclass testspider (scrapy. Spider): name= ' test ' download_delay=3 user_agent= ' mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/537.36 (khtml, like Gecko) chrome/45.0.2454.101 safari/537.36 ' Page_url = ' HTTP://WWW.TEST.COM/ZHAOP In/java/{0}/?filteroption=2 ' page=1 #执行入口 def start_requests (self): Yield scrapy #第一页.            Request (Self.page_url.format (' 1 '), headers={' user-agent ': self.user_agent}, Callback=self.parse, Errback=self.errback_httpbin) #解析返回的数据 def parse (self,response): For Li in Response.xpath ('//*[@id = "S_ Position_list "]/ul/li"): yield{' Company ': Li.xpath (' @data-company '). Extract (), ' s Alary ': Li.xpath (' @data-salary '). Extract ()} #是否是最后一页, based on the next page of the button CSS style to determine if RESPONSE.CSS (' A.PAGE_NO.P            Ager_next_disabled '): Print ('---is the last page,stop!---') Pass else: Self.page=self.page+1 #抓取下一页 yield scrapy. Request (Self.page_url.format (str (self.page)), headers={' user-agent ': self.user_agent}, Callba Ck=self.parse, Errback=self.errback_httpbin) #异常处理 def errback_httpbin (self,failure): if Failu         Re.check (httperror): Response = failure.value.response print ' httperror on {0} '. Format (Response.url) Elif Failure.check (dnslookuperror): request = failure.request print ' dnslookuperror on {0} '. f            Ormat (Request.url) elif Failure.check (Timeouterror, tcptimedouterror): request = Failure.request print ' Timeouterror on {0} '. Format (Request.url)

Start crawler: Scrapy runspider//spiders//test_spider.py-o test.csv After the completion of the file in CSV format:

Asynchronous paging

Page is partially refreshed when paging asynchronously, the URL address bar does not change

The data objects that the crawler parses are usually JSON

Test scenario: Capture a movie site's classic movie Top 100

#coding =utf-8import scrapyimport jsonclass testspider (scrapy. Spider): name = ' Test ' Download_delay = 3 User_agent = ' mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/537.36 (khtml, like Gecko) chrome/45.0.2454.101 safari/537.36 ' Pre_url = ' https://movie.douban.com/j    /search_subjects?type=movie&tag=%e7%bb%8f%e5%85%b8&sort=recommend&page_limit=20&page_start= ' Page=0 cnt=0 def start_requests (self): url= self.pre_url+str (0*20) yield scrapy. Request (url,headers={' user-agent ': self.user_agent},callback=self.parse) def parse (self,response): If Response.bo DY: # JSON string converted to Python object Python_obj=json.loads (response.body) subjects=python_obj[' Subjec                    TS '] If Len (subjects) >0:for sub in subjects:self.cnt=self.cnt+1                    Yield {' title ': sub["title"], ' rate ': sub["rate"]           }     If Self.cnt<100:print ' next page-------' self.page=self.page+1 Url= self.pre_url+str (self.page*20) yield scrapy. Request (url,headers={' user-agent ': self.user_agent},callback=self.parse)

Start crawler: Scrapy runspider//spiders//test_spider.py-o Test.json After the completion of a JSON-formatted file:

The difference between scrapy and BeautifulSoup or lxml

Scrapy is a complete set of frameworks for crawling and fetching data, and BeautifulSoup or lxml is just a library of parsing html/xml, functions like Scrapy's XPath and CSS selectors, so they can be used under Scrapy, but run less efficiently. With the Scrapy selector, we can copy the XPath and CSS values of any node directly using the browser's F12 mode.

