Python makes simple web crawlers, and python makes Crawlers

Python makes simple web crawlers, and python makes Crawlers

1. preparations:

To do something better, you must first sharpen the tool. Therefore, we need to configure a development environment suitable for ourselves before performing Coding. the development environment I set up is:

Operating System: Ubuntu 14.04 LTS
Python version: 2.7.6
Code Editor: Sublime Text 3.0

The background of this web crawler demand I plan to continue the demand of DotNet open source base camp in his article. I will not explain it here. We only capture the weather conditions of all major cities in a province from. This example uses Hubei province.
2. Actual Web Crawler:
2. 1. Get the city list:
First, we need to get the webpage of all cities in Hubei province, and then parse the webpage. Network Address: http://www.tianqihoubao.com/weather/province.aspx? Id = 420000
We can view the source code of this page and find that the list of all cities is based on <td style = "height: 22px "align =" center "> <a href =" city weather link + city name ">. Therefore, we can encapsulate a function to obtain the data we want by using a regular expression, the sample code is as follows:
Copy codeThe Code is as follows: def ShowCity ():
Html = requests. get ("http://www.tianqihoubao.com/weather/province.aspx? (Id = 420000 ")
Citys = re. findall ('<td style = "height: 22px" align = "center"> <a href = "(.*?) "> ', Html. text, re. S)
For city in citys:
Print city
The captured results are as follows:

1 top/anlu.html "title =" anlu historical weather query 2 top/badong.html "title =" Badong historical weather query 3 top/baokang.html "title =" Baokang historical weather query 4 top/caidian.html "title =" caidian historical weather query 5 top/changyang.html "title =" Changyang historical weather query 6 top/chibi.html "title =" Chibi historical weather query 7 top/chongyang.html "title =" chongyang historical weather query 8 top/dawu.html "title =" Dawu historical weather query 9 top/daye.html "title =" Daye historical weather Query 10 top/danjiangkou.html "title =" Danjiangkou historical weather Query 11 top/dangyang.html "title =" Dangyang historical weather Query 12 top/ezhou.html "title =" Ezhou historical weather query 13 top/enshi.html "title =" Enshi historical weather query 14 top/fangxian.html "title =" quxian historical weather query 15 top/gongan.html "title =" PSB historical weather query 16 top/gucheng.html "title =" Gucheng historical weather query 17 top/guangshui.html "title =" historical weather query 18 top/hanchuan.html "title =" Hanchuan historical weather query 19 top/hanyang.html "title =" Hanyang historical weather query 20 top/hefeng.html "title =" Hefeng historical weather Query 21 top/hongan.html "title =" hong'an historical weather query 22 top/honghu.html "title =" Hong Lake historical weather query 23 top/huangpi.html "title =" Huang Wei historical weather query 24 top/huanggang.html "title =" Huanggang historical weather query 25 top/huangmei.html "title =" Huangmei historical weather query 26 top/huangshi.html "title =" Huangshi historical weather query 27 top/jiayu.html "title =" jiayu historical weather query 28 top/jianli.html "title =" historical weather query 29 top/jianshi.html "title =" historical weather query 30 top/jiangxia.html "title =" Jiangxia historical weather Query 31 top/jingshan.html "title =" jingshan historical weather query 32 top/jingmen.html "title =" Jingmen historical weather query 33 top/jingzhou.html "title =" Jingzhou historical weather query 34 top/laifeng.html "title =" laifeng historical weather query 35 top/laohekou.html "title =" laohekou historical weather query 36 top/lichuan.html "title =" Lichuan historical weather query 37 top/lvtian.html "title =" roada historical weather query 38 top/macheng.html "title =" Macheng historical weather query 39 top/nanzhang.html "title =" Nanxun historical weather query 40 top/qichun.html "title =" Hunchun historical weather Query 41 top/qianjiang.html "title =" Qianjiang historical weather query 42 top/sanxia.html "title =" Three Gorges historical weather query 43 top/shennongjia.html "title =" Shennongjia historical weather query 44 top/shiyan.html "title =" Shiyan historical weather query 45 top/shishou.html "title =" Shi Shou historical weather query 46 top/songzi.html "title =" song Zi historical weather query 47 top/suizhou.html "title =" suizhou historical weather query 48 top/tianmen.html "title =" Tianmen historical weather query 49 top/hbtongcheng.html "title =" Tongcheng historical weather query 50 top/tongshan.html "title =" Tongshan historical weather Query 51 top/wufeng.html "title =" five peaks historical weather query 52 top/wuchang.html "title =" Wuchang historical weather query 53 top/wuhan.html "title =" Wuhan historical weather query 54 top/wuxue.html "title =" Wuxue historical weather query 55 top/hbxishui.html "title =" wushui historical weather query 56 top/xiantao.html "title =" Xiantao historical weather query 57 top/xianfeng.html "title =" xianfeng historical weather query 58 top/xianning.html "title =" Xianning historical weather query 59 top/xiangyang.html "title =" Xiangyang historical weather query 60 top/xiaogan.html "title =" Xiaogan historical weather Query 61 top/hbxinzhou.html "title =" Xinzhou historical weather query 62 top/xingshan.html "title =" Xingshan historical weather query 63 top/xuanen.html "title =" Xuan en historical weather query 64 top/hbyangxin.html "title =" Yangxin historical weather query 65 top/yiling.html "title =" Yiling historical weather query 66 top/yichang.html "title =" Yichang historical weather query 67 top/yicheng.html "title =" yicheng historical weather query 68 top/yidu.html "title =" Yidu historical weather query 69 top/yingcheng.html "title =" Yingcheng historical weather query 70 top/hbyingshan.html "title =" Yingshan historical weather Query 71 top/yuanan.html "title =" yuan'an historical weather query 72 top/yunmeng.html "title =" yunmeng historical weather query 73 top/yunxi.html "title =" Xi historical weather query 74 top/hbyunxian.html "title =" jinxian historical weather query 75 top/zaoyang.html "title =" Zaoyang historical weather query 76 top/zhijiang.html "title =" zhijiang historical weather query 77 top/zhongxiang.html "title =" zhongxiang historical weather query 78 top/zhushan.html "title =" ZHUSHAN historical weather query 79 top/zhuxi.html "title =" ZhuXi historical weather query 80 top/zigui.html "title =" historical weather Query 81 [Finished in 15.4 s]

. Obtain all weather information about the city:
Then we need to capture the weather conditions of the target City Based on the captured city link. Here we encapsulate a function to show all the weather conditions of the target city:

Def ShowWeather (city): res = str (city ). split ('"title ="') print res [1], '(daytime --> nighttime)' html = requests. get ("http://www.tianqihoubao.com/weather/00000 }". format (res [0]) weather = re. search ('<table width = "100%" border = "0" class = "B" cellpadding = "1" cellspacing = "1"> (. *?) </Table> ', html. text, re. S). group (1) res = re. findall (' <tr> (.*?) </Tr> ', weather, re. S) for x in res [2:]: w = re. findall ('> (.*?) <', X, re. s) for y in w [1:]: if len (y. strip () <= 0: pass else: print y print '--' * 40

In this way, we can get the weather conditions of the city !!

Complete code:

# Coding: UTF-8import reimport requestsimport sysreload (sys) sys. setdefaultencoding ('utf-8') def ShowWeather (city): res = str (city ). split ('"title ="') print res [1], '(daytime --> nighttime)' html = requests. get ("http://www.tianqihoubao.com/weather/00000 }". format (res [0]) weather = re. search ('<table width = "100%" border = "0" class = "B" cellpadding = "1" cellspacing = "1"> (. *?) </Table> ', html. text, re. S). group (1) res = re. findall (' <tr> (.*?) </Tr> ', weather, re. S) for x in res [2:]: w = re. findall ('> (.*?) <', X, re. s) for y in w [1:]: if len (y. strip () <= 0: pass else: print y print '--' * 40 print '\ n',' * 40def ShowCity (): html = requests. get ("http://www.tianqihoubao.com/weather/province.aspx? Id = 420000 ") citys = re. findall ('<td style =" height: 22px "align =" center "> <a href = "(.*?) "> ', Html. text, re. s) for city in citys: ShowWeather (city) def main (): ShowCity () if _ name __= = '_ main _': main ()

Yes, you are not mistaken. In just 34 lines of code, you can crawl all the weather conditions in all major cities in Hubei province in one month. Is it amazing !!??? But don't be happy too early. Everything has advantages and disadvantages. Let's take a look at its running results: [Finished in 371.8 s]

3. Knowledge summary:

3. 1. Encoding Problems:
# In ubuntu, due to encoding problems, we need to add a line of comment at the beginning of the Code to tell the Pyhton interpreter the encoding format we specified:

# In addition, we also need to set the default encoding format. Otherwise, Sublime Text cannot recognize Chinese characters and reports an error: "UnicodeEncodeError: 'ascii 'codec can't encode characters in position"

#-*-coding:utf8-*-import sysreload(sys)sys.setdefaultencoding('UTF-8')

3. 2. Regular Expression:

Import Regular Expression Library: import re
Match any character :.
Match the first character 0 times or unlimited times :*
Match the first character 0 times or once :?
Greedy Algorithm :.*
Non-Greedy Algorithm :.*?
Matching Number: (\ d +)
Common functions:

re.findall(pattern, string)re.search(pattern, string)re.sub(pattern, repl, string)

Finally, if you try to run the complete code I posted, you may encounter the same bottleneck as me, that is, the running speed is not fast enough (especially for computers with poor configuration like me ). It took 371.8 s to run this script on my machine. I have run it many times, and each time it is in the 350 +. Therefore, if your program does not care about the running speed, Python may be quite suitable. After all, you can use it to write less code to do more things !!!!