python利用beautifulSoup實現爬蟲

來源:互聯網
上載者:User
以前講過利用phantomjs做爬蟲抓網頁 www.jb51.net/article/55789.htm 是配合選取器做的

利用 beautifulSoup(文檔 :www.crummy.com/software/BeautifulSoup/bs4/doc/)這個python模組,可以很輕鬆的抓取網頁內容


# coding=utf-8import urllibfrom bs4 import BeautifulSoupurl ='http://www.baidu.com/s'values ={'wd':'網球'}encoded_param = urllib.urlencode(values)full_url = url +'?'+ encoded_paramresponse = urllib.urlopen(full_url)soup =BeautifulSoup(response)alinks = soup.find_all('a')

上面可以抓取百度搜出來結果是網球的記錄。

beautifulSoup內建了很多非常有用的方法。

幾個比較好用的特性:

構造一個node元素

代碼如下:

soup = BeautifulSoup('Extremely bold')tag = soup.btype(tag)#

屬性可以使用attr拿到,結果是字典

代碼如下:

tag.attrs# {u'class': u'boldest'}

或者直接tag.class取屬性也可。

也可以自由操作屬性


tag['class'] = 'verybold'tag['id'] = 1tag#Extremely bolddel tag['class']del tag['id']tag#Extremely boldtag['class']# KeyError: 'class'print(tag.get('class'))# None

還可以隨便操作,尋找dom元素,比如下面的例子

1.構建一份文檔


html_doc = """The Dormouse's storyThe Dormouse's storyOnce upon a time there were three little sisters; and their names wereElsie,Lacie andTillie;and they lived at the bottom of a well...."""from bs4 import BeautifulSoupsoup = BeautifulSoup(html_doc)

2.各種搞


soup.head#The Dormouse's storysoup.title#The Dormouse's storysoup.body.b# The Dormouse's storysoup.a# Elsiesoup.find_all('a')# [Elsie,# Lacie,# Tillie]head_tag = soup.headhead_tag#The Dormouse's storyhead_tag.contents[The Dormouse's story]title_tag = head_tag.contents[0]title_tag#The Dormouse's storytitle_tag.contents# [u'The Dormouse's story']len(soup.contents)# 1soup.contents[0].name# u'html'text = title_tag.contents[0]text.contentsfor child in title_tag.children:  print(child)head_tag.contents# [The Dormouse's story]for child in head_tag.descendants:  print(child)#The Dormouse's story# The Dormouse's storylen(list(soup.children))# 1len(list(soup.descendants))# 25title_tag.string# u'The Dormouse's story'
  • 聯繫我們

    該頁面正文內容均來源於網絡整理,並不代表阿里雲官方的觀點,該頁面所提到的產品和服務也與阿里云無關,如果該頁面內容對您造成了困擾,歡迎寫郵件給我們,收到郵件我們將在5個工作日內處理。

    如果您發現本社區中有涉嫌抄襲的內容,歡迎發送郵件至: info-contact@alibabacloud.com 進行舉報並提供相關證據,工作人員會在 5 個工作天內聯絡您,一經查實,本站將立刻刪除涉嫌侵權內容。

    A Free Trial That Lets You Build Big!

    Start building with 50+ products and up to 12 months usage for Elastic Compute Service

    • Sales Support

      1 on 1 presale consultation

    • After-Sales Support

      24/7 Technical Support 6 Free Tickets per Quarter Faster Response

    • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.