The use of Python data acquisition--beautifulsoup

The use of Python network data acquisition 1-beautifulsoup

来自此书: [美]Ryan Mitchell 《Python网络数据采集》, the example is copied, think that after knocking over or have a role, so record down.

Import requestsfrom bs4 Import beautifulsoupres = Requests.get (' https://www.pythonscraping.com/pages/page1.html ') Soup = BeautifulSoup (res.text, ' lxml ') print (SOUP.H1)

Using the Urllib access page, read returns the bytes that need to be decoded to the utf-8 text. Like this a.read().decode('utf-8') , however, when using BS4 parsing, you can directly pass in the response object returned by the Urllib library.

Import Urllib.requesta = Urllib.request.urlopen (' https://www.pythonscraping.com/pages/page1.html ') soup = BeautifulSoup (A, ' lxml ') print (SOUP.H1)

Crawl all the CSS class attributes to the green span tag, these are the names of people.

Import requestsfrom bs4 Import beautifulsoupres = Requests.get (' https://www.pythonscraping.com/pages/ Warandpeace.html ') soup = BeautifulSoup (res.text, ' lxml ') green_names = Soup.find_all (' span ', class_= ' green ') for name in Green_names:print (name.string)


Annapavlovna schererempress maryafedorovnaprince Vasili Kuraginanna pavlovnast. Petersburgthe Princeanna Pavlovnaanna Pavlovna ...
Children (child) and descendants (descendant) are not the same. The child tag is the immediate next generation of the parent tag, and the descendant tag includes all descendants below the parent tag. In layman's terms, descendant includes child.

Import requestsfrom bs4 Import beautifulsoupres = Requests.get (' https://www.pythonscraping.com/pages/page3.html ') Soup = BeautifulSoup (res.text, ' lxml ') gifts = soup.find (' table ', id= ' giftlist '). Childrenfor name in Gifts:print (name)


<tr><th>item title</th><th>description</th> <th>cost</th><th>image</th></tr><tr class= "Gift" id= "GIFT1" ><td> Vegetable Basket</td><td>this Vegetable Basket is the perfect gift for your health conscious (or overweight) f Riends!<span class= "Excitingnote" >now with Super-colorful Bell peppers!</span></td><td>$ 15.00</td><td></td></tr><tr class=" gift "id=" gift2 "><td>russian Nesting Dolls </td><td>hand-painted by trained monkeys, these exquisite dolls is priceless! And by "priceless," we mean "extremely expensive"! <span class= "Excitingnote" >8 entire dolls per set! Octuple the Presents!</span></td><td>$10,000.52</td><td></TD></TR> 
After finding the table, select the current node as TR and find the sibling node after the TR, since the first TR is the table header, this way, the text can extract all the body data except for the table header.

Import requestsfrom bs4 Import beautifulsoupres = Requests.get (' https://www.pythonscraping.com/pages/page3.html ') Soup = BeautifulSoup (res.text, ' lxml ') gifts = soup.find (' table ', id= ' giftlist '). Tr.next_siblingsfor name in Gifts:print (name)


<tr class= "Gift" id= "gift1" ><td>vegetable basket</td><td>this vegetable Basket is the perfect Gift for your health conscious (or overweight) Friends!<span class= "Excitingnote" >now with Super-colorful Bell Peppe Rs!</span></td><td>$15.00</td><td></td></tr><tr class=" gift "id=" gift2 "><td>russian Nesting Dolls </td><td>hand-painted by trained monkeys, these exquisite dolls is priceless! And by "priceless," we mean "extremely expensive"! <span class= "Excitingnote" >8 entire dolls per set! Octuple the Presents!</span></td><td>$10,000.52</td><td></td></tr>
Find the price of a product, you can find its parent tag according to the picture of the product <td> , the last sibling tag is the price.

Import requestsfrom bs4 Import beautifulsoupres = Requests.get (' https://www.pythonscraping.com/pages/page3.html ') Soup = BeautifulSoup (res.text, ' lxml ') Price = Soup.find (' img ', src= '). /img/gifts/img1.jpg '). Parent.previous_sibling.stringprint (Price)


$15.00
Collect all the pictures of the product, in order to avoid other pictures. Use regular expressions to refine your search.

Import reimport requestsfrom bs4 Import beautifulsoupres = Requests.get (' https://www.pythonscraping.com/pages/ Page3.html ') soup = BeautifulSoup (res.text, ' lxml ') imgs= soup.find_all (' img ', Src=re.compile (R '). /img/gifts/img.*.jpg ')) for the IMG in imgs:print (img[' src '))


.. /img/gifts/img1.jpg. /img/gifts/img2.jpg. /img/gifts/img3.jpg. /img/gifts/img4.jpg. /img/gifts/img6.jpg
find_all()You can also pass in a function, which requires that the return value must be a Boolean type, if True is preserved, and false if it is rejected.

Import reimport requestsfrom bs4 Import beautifulsoupres = Requests.get (' https://www.pythonscraping.com/pages/ Page3.html ') soup = BeautifulSoup (res.text, ' lxml ') # lambda tag:tag.name== ' img ' tags = soup.find_all (lambda tag:tag.has_ attr (' src ')) for tag in Tags:print (tag)



The tag is an element object that is has_attr used to determine whether the attribute is available. Tag.name is the name of the tag to get. In the page above, the following syntax returns the same result.
lambda tag: tag.has_attr('src')Orlambda tag: tag.name=='img'