Python crawls the weather forecast data and deposits it into local Excel __python

Recent sneak in, engaged in a few days Python crawler, basic can achieve conventional network data crawling, such as embarrassing encyclopedia, Watercress film review, NBA data, stock data, weather, etc. crawl, the whole process is actually relatively simple, there are some html+css+dom trees and other knowledge is easy, I'm going to take the weather forecast as an example and sort it out. requirements: Use Python to crawl the Weather network to specify the time period and region of weather forecast data, and to write the crawled data to the local Excel file in order. Environment configuration: python3.5 Win7 64-bit pandas (generated time series used) Urllib (get HTML) BeautifulSoup (parse HTML) xlsxwriter (write to Excel) Don't talk much, just do it. 1, the use of the library

#!/usr/bin/env Python3
#-*-coding:utf-8-*-
import datetime
Import pandas as PD
import Xlsxwriter as XL W from
urllib import request from
BS4 import BeautifulSoup as BS

2. Two methods of generating month interval

#  Method One: datetime, first convert to datetime type, then add and subtract
def daterange (start, end):  # start= ' 2014-09 '
    strptime, strftime = Datetime.datetime.strptime, datetime.datetime.strftime days
    = (Strptime (end, "%y-%m")-Strptime (Start, "%y-%m")) . Days
    Datelist1 = [Strftime (Strptime (Start, "%y-%m") + Datetime.timedelta (i), "%y%m") for I in range (0, days, 1)]
    Datelist = sorted (List (set (DATELIST1)) return
    datelist

# Method II: Pandas produces time series
def dateRange1 (start, end):
    datelist1 = [Datetime.datetime.strftime (x, '%y%m ') for X in List (Pd.date_range (Start=start, End=end))]
    Datelist = sorted (List (set (DATELIST1)) return
    datelist

3, crawl Weather Network data

You need to pay attention to the handling of the none value, and if none, there will be an error writing to local Excel because the str type is the default for writing, so the None value in the code is judged first and then the string is processed.

# Crawl Weather forecast
def getcommentsbyid (city, Start, end):  # City for string, year for list, month for list
    weather_result = []
    Datelist = DateRange (start, end) for
    I-datelist:
        url = ' http://lishi.tianqi.com/' + city + '/' + i + '. html ' 
  opener = Request. Request (URL)
        opener.add_header (' user-agent ', ' mozilla/4.0 (compatible; MSIE 5.5; Windows NT)
        req = Request.urlopen (opener). Read ()
        soup = BS (req, ' Html.parser ')
        weather_m = Soup.select ( ' Div. tqtongji2 > ul ')  #. Represents class; ' #tongji ' means the ID is equivalent to a[id= ' Tongji ']

        for I in Weather_m[1:]:  # Because the first is the header, the screen is removed
            TT = [] for
            J in range (6): C16/>t = I.find_all (' li ') [j].string
                if T not None:  # There is a none value for processing, otherwise it cannot be written to Excel
                    Tt.append (t)
                else:
                    tt.append (' None ')
            weather_result.append (TT) return
    Weather_result

4. Write crawled data to local Excel

The Xlsxwriter Library is responsible for writing data to Excel, which is highly efficient, supports streaming writes, and alleviates memory inefficiencies.

#  list data written to local Excel
def list_to_excel (weather_result, filename):
    workbook = xlw. Workbook (' e:\\%s.xlsx '% filename)
    sheet = workbook.add_worksheet (' weather_report ')
    title = [' Date ', ' highest temperature ', ' Minimum temperature ', ' weather ', ' wind Direction ', ' wind '] for
    i in range (len (title)):
        sheet.write_string (0, I, Title[i], Workbook.add_format ({' Bold ': True})  # Written to header, font bold
    row, col = 1, 0 for
    A, B, C, D, E, F in Weather_result:
        sheet.write_string (row , col, a)
        sheet.write_string (Row, col + 1, b)
        sheet.write_string (Row, col + 2, c)
        sheet.write_string (Row, Col + 3, D)
        sheet.write_string (Row, col + 4, E)
        sheet.write_string (Row, col + 5, F)
        row + + 1
    workbook. Close ()

5, the finished, directly to the E interrogation to see

if __name__ = = ' __main__ ':
    data = Getcommentsbyid (' Jinan ', ' 2016-05 ', ' 2017-07 ')
    list_to_excel (data, ' Jinan Weather 201605-201707 ')

footnotes

If there is a mistake, please correct the communication.
The hoof walked swiftly and steadily, and put it on the line, and practiced the word. -zlg358 2017/8/23 a.m.