以前講過利用phantomjs做爬蟲抓網頁 www.jb51.net/article/55789.htm 是配合選取器做的
利用 beautifulSoup(文檔 :www.crummy.com/software/BeautifulSoup/bs4/doc/)這個python模組,可以很輕鬆的抓取網頁內容
# coding=utf-8import urllibfrom bs4 import BeautifulSoupurl ='http://www.baidu.com/s'values ={'wd':'網球'}encoded_param = urllib.urlencode(values)full_url = url +'?'+ encoded_paramresponse = urllib.urlopen(full_url)soup =BeautifulSoup(response)alinks = soup.find_all('a')
上面可以抓取百度搜出來結果是網球的記錄。
beautifulSoup內建了很多非常有用的方法。
幾個比較好用的特性:
構造一個node元素
代碼如下:
soup = BeautifulSoup('Extremely bold')tag = soup.btype(tag)#
屬性可以使用attr拿到,結果是字典
代碼如下:
tag.attrs# {u'class': u'boldest'}
或者直接tag.class取屬性也可。
也可以自由操作屬性
tag['class'] = 'verybold'tag['id'] = 1tag#Extremely bolddel tag['class']del tag['id']tag#Extremely boldtag['class']# KeyError: 'class'print(tag.get('class'))# None
還可以隨便操作,尋找dom元素,比如下面的例子
1.構建一份文檔
html_doc = """The Dormouse's storyThe Dormouse's storyOnce upon a time there were three little sisters; and their names wereElsie,Lacie andTillie;and they lived at the bottom of a well...."""from bs4 import BeautifulSoupsoup = BeautifulSoup(html_doc)
2.各種搞
soup.head#The Dormouse's storysoup.title#The Dormouse's storysoup.body.b# The Dormouse's storysoup.a# Elsiesoup.find_all('a')# [Elsie,# Lacie,# Tillie]head_tag = soup.headhead_tag#The Dormouse's storyhead_tag.contents[The Dormouse's story]title_tag = head_tag.contents[0]title_tag#The Dormouse's storytitle_tag.contents# [u'The Dormouse's story']len(soup.contents)# 1soup.contents[0].name# u'html'text = title_tag.contents[0]text.contentsfor child in title_tag.children: print(child)head_tag.contents# [The Dormouse's story]for child in head_tag.descendants: print(child)#The Dormouse's story# The Dormouse's storylen(list(soup.children))# 1len(list(soup.descendants))# 25title_tag.string# u'The Dormouse's story'