Common Python crawler modules, BeautifulSoup notes, and beautifulsoup Crawlers

Common Python crawler modules, BeautifulSoup notes, and beautifulsoup Crawlers

Import urllib. request as requestimport refrom bs4 import * # url = 'HTTP: // zh.house.qq.com/'url = 'HTTP: // www.0756fang.com/'html = request. urlopen (url ). read (). decode ('utf-8') soup = BeautifulSoup (html, "html. parser ") print (soup. head. meta ['content']) # print (soup. span. string); print (soup. span. text) # The results are the same. The text # name attribute of the returned Tag is the <ResultSet> class of the ''Tag. It is a listprint (soup) composed of <Tag>. find_all (attrs = {'Name': 'keyword'}) print (soup. find_all (class _ = 'site _ name') # The class attribute is the list of ''<Tag>, that is, <ResultSet> print (soup. find_all (class _ = 'site _ name') [0]) # This is a <Tag> print (soup. find (attrs = {'name': 'keyword'}) # The <Tag> class print (soup. find ('meta', attrs = {'name': 'keyword'}) # The name attribute is the print (soup) class of the ''meta Tag. find ('meta', attrs = {'name': 'keyword'}) ['content']) # <Tag class> You can directly query attribute values # used with the re module, case sensitivity can be ignored # as shown in the following example, you can find that the name attribute is key Meta tag print (soup. find ('meta', attrs = {'name': re. compile ('keyword', re. IGNORECASE)}) '''alipay''' ------------------------------ modify BeautifulSoup --------------------------------- ''' then '''soup. find (attrs = {'name': 'keyword '}). extract # Call this method to delete this A tag soup. title. name = 'ppp '# You can change the Tag name <title> to <ppp> # append (), insert (), insert_after (), or insert_before () to insert the new tag. Tag1 =. new_tag ('lil', class _ = '000000') ''' creates a Tag '''. title. append (Tag1) # Add Tag1 as the last [subnode] of the Tag whose name is the title, without line breaks #. insert (0, Tag1) ---- if insert is used here, the first parameter can control the priority of the added subnode #. insert_after (Tag1) --- same as insert_before, add it as the [sibling node] soup of the Title. head. meta ['content'] = 'You can enter (or change) the Tag's content attribute (value) 'del soup. head. meta ['content'] # This syntax can directly Delete the content attribute soup of this Tag. li. clear # Call the method to clear textsoup of all li labels. title. string = 'use this method to modify the title tag content' # use it with caution. It is only used for the smallest child node and the parent node clears the child node soup. div. append ('put at the end of the last append of the div subnode and the content of the label') soup. div. insert (0, 'Put at the beginning of the div subnode insert [0] before, is the TAG content '')

This is my previous BS4 note. Please contact QQ 328123440