Python crawler Series (vii): Improve analytic efficiency

It is a waste of memory and time to parse the entire document just because you want to find the <a> tag in the document. The quickest way is to ignore the <a> tag from the start. The Soupstrainer class can define a section of a document so that it does not have to parse the entire document first, only the documents defined in Soupstrainer. Create a Soupstrainer object and give BeautifulSoup a constructor as the parse_only parameter

Target Document

From BS4 import BeautifulSoup, navigablestring
From BS4 import Soupstrainer

Only_a_tags = Soupstrainer ("a")
ONLY_TAGS_WITH_ID_LINK2 = Soupstrainer (id= "Link2")


def is_short_string (String):
Return Len (String) < 6


Only_short_strings = Soupstrainer (text=is_short_string)

Soup = BeautifulSoup (Html_doc, "lxml")
Print (' 1------------Find all a elements ')
Print (BeautifulSoup (Html_doc, "Html.parser", Parse_only=only_a_tags). Prettify ())
Print (' 2------------find elements of Id=link2 ')
Print (BeautifulSoup (Html_doc, "Html.parser", Parse_only=only_tags_with_id_link2). Prettify ())
Print (' 3------------found element with element length less than 10 ')
Print (BeautifulSoup (Html_doc, "Html.parser", parse_only=only_short_strings). Prettify ())

Today, the foundation of our reptile series is even over. These are the tutorials for beginners in the company's training. In fact, in the actual process of the project, there are too many pits, the knowledge base is gradually shared behind the things.

Code word is not easy, look forward to praise.

Python crawler Series (vii): Improve analytic efficiency