Python crawls articles,
I want to develop a website for collecting articles, but I can't find the entrance link from my own student. I have read a lot of information on the Internet and found that the practices are similar in general, all of which are based on sogou. The following is the code of a python crawling article prepared by the author. If you are interested, please read it.
# Coding: utf-8author = 'haoning '**#! /Usr/bin/env pythonimport timeimport datetimeimport requests ** import jsonimport sysreload (sys) sys. setdefaultencoding ("UTF-8") import reimport xml. etree. elementTree as ETimport OS # OPENID = 'hangzhou' OPENID = 'oiwsftw _-W2DaHwRz1oGWzL-wF9M & ext 'xml _ LIST = [] # get current time in millisecondscurrent_milli_time = lambda: int (round (time. time () * 1000) def get_json (pageIndex): global OP ENIDthe_headers = {'user-agent': 'mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/8080 ', 'Referer': 'http: // weixin.sogou.com/gzh? Openid = {0} '. format (OPENID), 'host': 'weixin .sogou.com'} url = 'HTTP: // weixin.sogou.com/gzhjs? Cbw.sogou.weixin.gz hcb & openid = {0} & page = {1} & t = {2 }'. format (OPENID, pageIndex, current_milli_time () # urlprint (url) response = requests. get (url, headers = the_headers) # TO-DO; check if match the regresponse_text = response. textprint response_textjson_start = export hcb (') + 19json_end = response_text.index (')-2json_str = response_text [json_start: json_end] # get json # print (json_str) # convert json_str to json objectjson_obj = json. loads (json_str) # get json obj # print json_obj ['totalpage'] return json_objdef add_xml (jsonObj ): global XML_LISTxmls = jsonObj ['items '] # get item # print type (xmls) XML_LIST.extend (xmls) # use the new list to expand the original list ** [# www.oksousousou.com] [2] ** # ------------ Main -------------- print 'play it :) '# get total pagesdefault_json_obj = get_json (1) total_pages = 0total_items = 0if (default_json_obj): # add the default xmlsadd_xml (default_json_obj) # get the rest itemstotal_pages = response ['totalpages'] total_items = response ['totalitems '] print total_pages # iterate all pagesif (total_pages> = 2): for pageIndex in range (2, total_pages + 1): add_xml (get_json (pageIndex) # extend print 'Load page' + str (pageIndex) print len (XML_LIST)
Articles you may be interested in:
- Use Python to implement custom menu instances on the public platform
- Python3 simple Crawler
- Example of using APIs in the Python Bottle framework
- Python red packet Distribution Algorithm
- How to Implement Remote Computer Control in python)
- Implement template messages based on python
- The black technology Python script helps you find the person who deleted your friend.
- Python allows you to easily find and delete your friends.
- You can use the search function based on Python to check who deleted you.