Pyspider example code two: parsing JSON data

This series of articles mainly records and explains Pyspider's sample code, hoping to introduce. The official website of the Pyspider sample code is http://demo.pyspider.org/. The example code above is too much to do. So I find a more classic example of a simple explanation, I hope to have some help for the novice.

Example Description:

The contents of the Pyspider crawl are returned through the callback's parameter response, and response has several parsing methods.
1. Response.json for parsing JSON data
2, Response.doc returned is the Pyquery object
3, Response.etree returned is the lxml object
4. Response.text returns Unicode text
5, Response.content return is the byte code
This example mainly uses Response.json to parse the returned JSON data. Examples of other return types are shown in subsequent articles.

How to use:

Example code:

#!/usr/bin/env python#-*-encoding:utf-8-*-#Created on 2016-06-21 13:57:13#Project:duitang fromPyspider.libs.base_handlerImport*classHandler (Basehandler): Crawl_config={} @every (minutes=24 * 60)    defOn_Start (self): Self.crawl ('http://www.duitang.com/napi/friendship/fans/?start=0&limit=1000&user_id=116965', callback=self.index_page) @config ( age=10 * 24 * 60 * 60)    defindex_page (Self, Response): foreachinchresponse.json['Data']['object_list']: ID= each['ID'] Self.crawl ('http://www.duitang.com/napi/friendship/fans/?start=0&limit=1000&user_id='+str (ID), callback=self.index_page) Self.crawl ('http://www.duitang.com/napi/people/profile/?user_id='+str (ID), callback=self.detail_page) Start= response.json['Data']['Next_start'] Total= response.json['Data'][' Total'] User= response.json['Data']['Visit_user']['user_id']        ifStart <Total:self.crawl ('http://www.duitang.com/napi/friendship/fans/?start='+str (start) +'&limit=1000&user_id='+STR (user), callback=self.index_page) @config ( priority=2)    defdetail_page (Self, Response):return {            "username": response.json['Data']['username'],             "ID": response.json['Data']['ID']        }

Pyspider example code two: parsing JSON data