The use of Python data acquisition--beautifulsoup

Source: Internet
Author: User

The use of Python network data acquisition 1-beautifulsoup

来自此书: [美]Ryan Mitchell 《Python网络数据采集》, the example is copied, think that after knocking over or have a role, so record down.

Import requestsfrom bs4 Import beautifulsoupres = Requests.get (' https://www.pythonscraping.com/pages/page1.html ') Soup = BeautifulSoup (res.text, ' lxml ') print (SOUP.H1)

Using the Urllib access page, read returns the bytes that need to be decoded to the utf-8 text. Like this a.read().decode('utf-8') , however, when using BS4 parsing, you can directly pass in the response object returned by the Urllib library.

Import Urllib.requesta = Urllib.request.urlopen (' https://www.pythonscraping.com/pages/page1.html ') soup = BeautifulSoup (A, ' lxml ') print (SOUP.H1)

Crawl all the CSS class attributes to the green span tag, these are the names of people.

Import requestsfrom bs4 Import beautifulsoupres = Requests.get (' https://www.pythonscraping.com/pages/ Warandpeace.html ') soup = BeautifulSoup (res.text, ' lxml ') green_names = Soup.find_all (' span ', class_= ' green ') for name in Green_names:print (name.string)


Annapavlovna schererempress maryafedorovnaprince Vasili Kuraginanna pavlovnast. Petersburgthe Princeanna Pavlovnaanna Pavlovna ...

Children (child) and descendants (descendant) are not the same. The child tag is the immediate next generation of the parent tag, and the descendant tag includes all descendants below the parent tag. In layman's terms, descendant includes child.

Import requestsfrom bs4 Import beautifulsoupres = Requests.get (' https://www.pythonscraping.com/pages/page3.html ') Soup = BeautifulSoup (res.text, ' lxml ') gifts = soup.find (' table ', id= ' giftlist '). Childrenfor name in Gifts:print (name)


<tr><th>item title</th><th>description</th> <th>cost</th><th>image</th></tr><tr class= "Gift" id= "GIFT1" ><td> Vegetable Basket</td><td>this Vegetable Basket is the perfect gift for your health conscious (or overweight) f Riends!<span class= "Excitingnote" >now with Super-colorful Bell peppers!</span></td><td>$ 15.00</td><td></td></tr><tr class=" gift "id=" gift2 "><td>russian Nesting Dolls </td><td>hand-painted by trained monkeys, these exquisite dolls is priceless! And by "priceless," we mean "extremely expensive"! <span class= "Excitingnote" >8 entire dolls per set! Octuple the Presents!</span></td><td>$10,000.52</td><td></TD></TR> 

After finding the table, select the current node as TR and find the sibling node after the TR, since the first TR is the table header, this way, the text can extract all the body data except for the table header.

Import requestsfrom bs4 Import beautifulsoupres = Requests.get (' https://www.pythonscraping.com/pages/page3.html ') Soup = BeautifulSoup (res.text, ' lxml ') gifts = soup.find (' table ', id= ' giftlist '). Tr.next_siblingsfor name in Gifts:print (name)


<tr class= "Gift" id= "gift1" ><td>vegetable basket</td><td>this vegetable Basket is the perfect Gift for your health conscious (or overweight) Friends!<span class= "Excitingnote" >now with Super-colorful Bell Peppe Rs!</span></td><td>$15.00</td><td></td></tr><tr class=" gift "id=" gift2 "><td>russian Nesting Dolls </td><td>hand-painted by trained monkeys, these exquisite dolls is priceless! And by "priceless," we mean "extremely expensive"! <span class= "Excitingnote" >8 entire dolls per set! Octuple the Presents!</span></td><td>$10,000.52</td><td></td></tr>

Find the price of a product, you can find its parent tag according to the picture of the product <td> , the last sibling tag is the price.

Import requestsfrom bs4 Import beautifulsoupres = Requests.get (' https://www.pythonscraping.com/pages/page3.html ') Soup = BeautifulSoup (res.text, ' lxml ') Price = Soup.find (' img ', src= '). /img/gifts/img1.jpg '). Parent.previous_sibling.stringprint (Price)


$15.00

Collect all the pictures of the product, in order to avoid other pictures. Use regular expressions to refine your search.

Import reimport requestsfrom bs4 Import beautifulsoupres = Requests.get (' https://www.pythonscraping.com/pages/ Page3.html ') soup = BeautifulSoup (res.text, ' lxml ') imgs= soup.find_all (' img ', Src=re.compile (R '). /img/gifts/img.*.jpg ')) for the IMG in imgs:print (img[' src '))


.. /img/gifts/img1.jpg. /img/gifts/img2.jpg. /img/gifts/img3.jpg. /img/gifts/img4.jpg. /img/gifts/img6.jpg

find_all()You can also pass in a function, which requires that the return value must be a Boolean type, if True is preserved, and false if it is rejected.

Import reimport requestsfrom bs4 Import beautifulsoupres = Requests.get (' https://www.pythonscraping.com/pages/ Page3.html ') soup = BeautifulSoup (res.text, ' lxml ') # lambda tag:tag.name== ' img ' tags = soup.find_all (lambda tag:tag.has_ attr (' src ')) for tag in Tags:print (tag)


The tag is an element object that is has_attr used to determine whether the attribute is available. Tag.name is the name of the tag to get. In the page above, the following syntax returns the same result.
lambda tag: tag.has_attr('src')Orlambda tag: tag.name=='img'

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.