"Python Network data Acquisition" Reading notes (ii)

1. Find labels by name and attributes

As before, grab the entire page and create a BeautifulSoup object. This "lxml" parser needs to be downloaded separately.

PIP3 Install lxml

>>> from urllib.request import urlopen>>> from BS4 import beautifulsoup>>> html = Urlopen ("htt P://www.pythonscraping.com/pages/warandpeace.html ") >>> bsobj = BeautifulSoup (HTML," lxml ")

Finall () can get all the specified labels on the page, extract only the text contained in the <span class= "green" ></span> tags, and get a list of people names. Get_text () Clears all the labels and returns a string containing only the text.

>>> namelist = Bsobj.findall ("span", {"Class": "Green"}) >>> for name in Namelist:print (name.get_t EXT ())

2. Detailed parameters of Finall () and find ()

FINDALL (tag, attributes, recursive, text, limit, keywords)

Find (tag, attributes, recursive, text, keywords)

tag--can pass a label name or a list of multiple label names to make parameters; The following example returns a list that contains all the title labels in the HTML document

>>> Bsobj.findall ({"H1", "H2", "H3", "H4", "H5", "H6"}) [

The attributes--dictionary encapsulates several properties and corresponding property values for a label, and the following example returns a span label for red and green colors in an HTML document
>>> Bsobj.findall ("span", {"class": {"green", "Red"}})

recursive--Boolean variable (default true) to find all child tags of the label parameter, and sub-labels of child labels; false to find only the first-level labels of the document

text--is used to match the text content of the label, and the following example is to find the number of tags in the page that contain "the Prince" content
>>> Namelist=bsobj.findall (text= "the Prince") >>> print (len (namelist)) 7

limit--the first X results (equal to 1 o'clock equivalent to find) in the order of the pages

keywords--allows you to select the labels with the specified attributes
>>> Alltext = Bsobj.findall (id= "text") >>> print (Alltext[0].get_text ())
The following two lines of code are exactly the same
Bsobj.findall (id= "text") Bsobj.findall ("", {"id": "Text"})
Because class is a keyword, you need to add an underscore when you specify with keywords
Bsobj.findall (class_= "green") Bsobj.findall ("", {"Class": "Green"})

3. BeautifulSoup Object
BeautifulSoup object: As in the previous code example, Bsobj
Tag object: A column image or a single object (such as BSOBJ.DIV.H1) obtained by the BeautifulSoup object through find and findall or directly called child tags
Navigablestring object: Represents the text in the label
Comment objects: Finding comment labels for HTML documents

4. Navigation of the Tag parsing tree: Find the label by its position in the document.

Working with child tags and descendants tags:
Child tags are the next level of a parent tag.
Descendant tags (descendant) are labels for all levels below a parent tag.
The same
>>> from urllib.request import urlopen>>> from BS4 import beautifulsoup>>> html = Urlopen ("htt P://www.pythonscraping.com/pages/page3.html ") >>> bsobj = BeautifulSoup (HTML," lxml ")
Just want to find sub-tags, you can use. Children; The following example prints data rows for all products in the Giftlist table (for example, using. Descendants will print out more than 20 labels)
>>> for child in Bsobj.find ("table", {"id": "giftlist"}). Children:print (Child)

Deal with brother Tags:
Next_siblings and Previous_siblings will return a set of labels
Next_sibling and Previous_sibling will return a single label
The following example prints products for all rows except the first row in the product list
>>> for sibling in Bsobj.find ("table", {"id": "giftlist"}). Tr.next_siblings:print (sibling)

Working with parent Tags:
Parent and parents
The following example prints the price of the item corresponding to the specified picture.
>>> Print (Bsobj.find ("img", {"src": "). /img/gifts/img1.jpg "}). Parent.previous_sibling.get_text ()) $15.00

"Python Network data Acquisition" Reading notes (ii)