BeautifulSoup: Function usage

# -*- coding: utf-8 -*-

' # Author:solomon Xie # Usage: Test BeautifulSoup Some usage and bug-prone places # Enviroment:python 2.7, Windows 7 (32 bit), Chinese Language Pack "Import time, reImport BS4# must be imported because of the need to do some BS4 proprietary types of judgmentFrom BS4Import BeautifulSoupDefTest_beautifulsoup():"" # Some of BeautifulSoup's problems really make people egg big. # Here's the study. """' # Basic Part # about BS4 parsing Speed ################# # After reading the document carefully, the document parser is critical to speed! # If the Cchardet module is not installed, then light a webpage will be 7 seconds!! # does not include time to get pages. However, after the test, such as roller coaster: # installed Cchardet after 7 seconds into a moment. # However, after a few days and then changed back to 7 seconds, unloading the cchardet and changed back to a moment! # In addition, BeautifulSoup upgrade to 4, the import method has changed, as follows: "From BS4Import BeautifulSoup"# about the encoded format of the parsed document ########## # and the official said that no matter what encoded document is passed in, it will be unified into Unicode # In fact sometimes I find it necessary to pass in Unicode to get the correct results ... # The experiment here finds that it's really true! Must pass in the decode code "' Html_doc = open (' Test-zhilian-list-page-sm1.html ',' R '). Read (). Decode (' Utf-8 ')# ^ This HTML file is actually the source code of the Zhaopin search page, you can save it and try it out directly."# about BS4 's document Parser ############## # is another big pit: After the BS upgrade to 4, you need to explicitly specify the document parser when instantiating, such as: # soup = BeautifulSoup (Html_doc, ' lxml ') # but the famous lxml in This is a big hole, # because it will skip the HTML all do not write the standard tag, and no matter how much people care about the information # because of this parser, I said less also toss for several hours to find the reason it. # Summary: Remember, choose html5lib! Efficiency did not check how much, at least the fault-tolerant rate is strong, will not delete your things! "' Soup = BeautifulSoup (Html_doc,' Html5lib ')"# about BS4 's output format ################# # prettify () official explanation is to output Utf-8 format, # is actually Unicode type!! Therefore, you must specify the encoding in the Prettify (). ‘‘‘# output = soup.prettify (' Utf-8 ')# print repr (output)The so-called multiple search node way ############## just don't know why: no matter how the quiz, find () and Find_all () is dead or alive does not work! Only the English version of "Alice" in the official document will be tested. In other words, the problem is still on the text encoding? But when I try to find English, the search results are still zero-_-! At the end of the BS4, the only thing that can be used in the Search tool is select (), which is the CSS selector. Although it is extremely useful, it is still restrictive. Do not give up, so I still try again Find_all () of the problem. ‘‘‘# = = Find_all () search tag name ============ OK# result = Soup.find_all (' DL ') # OK# = = Find_all () The search Tag property ============ not all OK# result = Soup.find_all (id= ' Newlist_list_div ') # OK# result = Soup.find_all (Href=re.compile ('. htm ')) # Failed actually doesn't support href search, and the official said it's not the same# result = Soup.find_all (name= ' Vacancyid ') # Failed does not support label Name property Search# = = Find_all () by CSS search ============ OK# result = Soup.find_all (' div ', class_= ' Clearfix ') # OK# result = Soup.find_all (' div ', class_=re.compile (' Newlist_detail ')) # OK# result = Soup.find_all (Class_=re.compile (' Newlist_detail ')) # OK# = = Find_all () by content text Search ============# Find_all () plus the text parameter,# The string is returned! And not tag!!.# type: <class ' bs4.element.NavigableString ' ># result = Soup.find_all (text= ' accountant ') # OK content must be exactly equal to count! (No sub-labels included)# result = Soup.find_all (text=u ' data ') # OK content must be exactly equal doesn't matter Unicode# result = Soup.find_all (text=re.compile (U ' Education: ')) # OK Unicode is absolutely wanted! Or not!# = = Select (), CSS selector search engine ============"CSS selector syntax see W3cschool's Documentation: http://www.w3school.com.cn/cssref/selector_nth-of-type.asp The following is a summary of the syntax search in BeautifulSoup: Tag search, such as ' input ', to search all the elements of a broad path labeled input, such as: ' Body a ', which is the absolute path to all a elements within the body, such as: ' Body > div > div & Gt P ', must be fully compliant with the path to search for ID, such as: ' #tag-1 ', search for a label with ID TAG1 search, such as: ' Div #tag1 ', search for the div tag with id xx ' div[class*=newlist_detail] ~ div[ Class*=newlist_detail] ', a large mixed attribute exists, such as: ' A[href ', searches for all a-label class names that have href attributes, such as: ' [Class=clearfix] ', to find a label with class name equal to Clearfix ' [ Class^=newlist_detail] ', find the tag ' [CLASS$=ZWMC] ' in class name that begins with ' Newlist_detail ', and find the label ' [Class*=clearfix] ' ending with ' ZWMC ' in the class name ', find the tag brother search with "ZWMC" in the class name, such as: ' #links ~. Clearfix ', and find all the classes that have the ID of the links tag ' brother tag ' #links +. Clearfix ' for the ' Clearfix ', Find the next class ID for the links tag is equal to "clearfix" Brother tag sequence search, such as: ' P Nth-of-type (3) ', this plainly is to choose the 3rd P tag ' P Nth-of-type (odd) ' means the odd P label ' p Nth-of-type (even) ' represents the even P label ' P nth-of-type (n) ' means that all P-tags ' p nth-of-type (3n) ' represents a multiples of 3 p-tags ' P-nth-of-type (4n+1) ' Represents a multiple of 4 plus 1 p tags, such as 5th, 9th "# result = Soup.select (' DL > P ') # OK tag Path search# result = Soup.select (' div[class*=newlist_detail] ~ div ') # OK various hybrid search# result = Soup.select (' [CLASS*=ZWMC] ') # OK various mixed search con = soup.select (' Div[class^=newlist_detail] ') [0] result = Con.select (' [CLASS*=ZWMC] ')# Print type (result[0])Print Len (Result)# out = Soup.select (' [CLASS*=ZWMC] ')# Print Len (out)# for item in out:# print Item.get_text (). Encode (' Utf-8 ')DefBstext(tags=[], info=‘‘):If Len (tags): t = tags[0]# because there's only one objectThe # Select () selector returns the tag tag# while Find_all () is returned with a text query is a string!If Isinstance (T, Bs4.element.Tag):return t.get_text (). Encode ( ' utf-8 ') elif isinstance (T, bs4.element.NavigableString): return T.string.encode ( ' utf-8 ') else:  return # Compute time < Span class= "Hljs-keyword" >def timeup (func): start = Time.clock () Func () end = Time.clock () timeuse = End-start print  ' \n[%s ()] The function uses%d seconds altogether. \ n '% (func.__name__, timeuse) return timeuseif __name__ = = Span class= "hljs-string" > ' __main__ ': Timeup (test_beautifulsoup)        

BeautifulSoup: Feature use