Scraper——BeautifulSoup and LXML，beautifulsouplxml

Scraper——BeautifulSoup and LXML，beautifulsouplxml

爬蟲解析方式除了Regex，還有BeautifulSoup包和LXML模組。現在分別來介紹這兩種方式。
1.BeautifulSoup包
功能比Regex很多，且要簡潔明白一些。但是，由於它是用python編寫出來的包，速度會慢一些。

# 資料抓取——BeautifulSoup包'''官方文檔：https://www.crummy.com/software/BeautifulSoup/bs4/doc.zh/'''# beautifulsoup包處理錯誤的HTML格式from bs4 import BeautifulSoupbroken_html = '<ul class=country><li>Area<li>Population</ul>'soup = BeautifulSoup(broken_html, "html.parser")fixed_html = soup.prettify()# 修複HTML格式# print fixed_htmlul = soup.find('ul', attrs={'class': 'country'})# 調取元素# print ul.find('li')# print ul.find_all('li')# 現在用此方法抽取國家面積資料import urllib2def download(url, user_agent="wswp", num_retries=2):    print "Download :", url    headers = {"User_agent": user_agent}    request = urllib2.Request(url, headers=headers)    try:        html = urllib2.urlopen(request).read()    except urllib2.URLError as e:        print "Download Error :", e.reason        html = None        if num_retries > 0:            if hasattr(e, "code") and 500 <= e.code < 600:                return download(url, user_agent, num_retries-1)    return htmlif __name__ == "__main__":    url = "http://example.webscraping.com/view/United-Kingdom-239"    html = download(url)    soup = BeautifulSoup(html, "html.parser", from_encoding="utf-8")    # 先找到其父元素    tr = soup.find(attrs={'id':'places_area__row'})    # 然後再找到面積所在的子項目    td = tr.find(attrs={'class':'w2p_fw'})    # 最後輸出子項目的內容    area = td.text    print area# 總結：BeautifulSoup包雖然比Regex要複雜，但是，並不難懂，而且更易構造和理解。最後，像多餘的空格和標籤屬性這種布局上的小變化，我們使用BeautifulSoup包更為方便。

2.LXML模組

這此模組中有一個CSS選取器。在使用前，必須先要安裝cssselect包。不然，會出現錯誤！

# 資料抓取——Lxml模組'''Lxml是基於libxml2這一XML解析庫的Python封存，該模組的解析速度更加塊，比BeautifulSoup包快，因為，它使用的C語言編寫。'''# 使用第一步先將不合法的HTML解析為統一的格式。import lxml.htmlimport urllib2'''broken_html = '<ul class=country><li>Area<li>Population</ul>'# 解析htmltree = lxml.html.fromstring(broken_html)fixed_html = lxml.html.tostring(tree, pretty_print=True)'''# print fixed_htmldef download(url, user_agent="wswp", num_retries=2):    print "Download :", url    headers = {"User_agent": user_agent}    request = urllib2.Request(url, headers=headers)    try:        html = urllib2.urlopen(request).read()    except urllib2.URLError as e:        print "Download Error :", e.reason        html = None        if num_retries > 0:            if hasattr(e, "code") and 500 <= e.code < 600:                return download(url, user_agent, num_retries - 1)    return htmlif __name__ == "__main__":    url = "http://example.webscraping.com/view/United-Kingdom-239"    html = download(url)    tree = lxml.html.fromstring(html)    td = tree.cssselect('tr#places_area__row > td.w2p_fw')[0]  # 注意在最新的lxml模組中已經沒有cssselect包，需要單獨下載 pip install cssselect    area = td.text_content()    print area