Python sgmlparser learning experience

PythonSgmlparserIt is very convenient for the module to process HTML Parsing. It divides HTML processing into three steps: Breaking HTML into its components, processing the fragments, and then merging the fragments into HTML. The first step is done through sgmllib. py, which is part of the standard Python library.

The key to understanding this chapter is to know that HTML is not just text, but also structured text. This structure comes from the more or less hierarchical sequence of start and end tags. Generally, you do not process HTML in this way, but process it in text editing, you can also use a browser to browse or edit pages in a visual manner. Sgmllib. py shows the HTML structure.

Sgmllib. py contains an important class: sgmlparser. Sgmlparser splits HTML into useful fragments, such as start tag and end tag. After it successfully breaks down a piece of data into a useful segment, it calls its own internal Method Based on the discovered data. To use this analyzer, You need to subclass the sgmlparser class and override these methods. This is what I mean when I say it represents the HTML structure: the HTML structure determines the order of method calls and the parameters passed to each method.

Sgmlparser analyzes HTML into eight types of data, and then calls a separate method for each type:

Start tag)
Is the HTML tag of the starting block, such as, or

Or a unique tag,
Image
Or
. When it finds a starting tagname,
Sgmlparser searches for methods named start_tagname or do_tagname.
For example, when it finds
Mark, it will find a start_pre or do_pre method.
If it is found, sgmlparser will use the attribute list of this tag to call this method;
Otherwise, it uses the name and attribute list of the tag to call the unknown_starttag method.
End tag)
Is to end the HTML Tag of a block, such as, or
 
 
.
 
 
 
 
 
When an end tag is found, sgmlparser searches for the method named end_tagname.
 
 
If found, sgmlparser calls this method, otherwise it uses the name of the tag to call unknown_endtag.
 
 
Character reference)
 
 
Escape characters expressed in decimal or equivalent hexadecimal notation,
 
 
Image. When found, sgmlparser calls handle_charref using a decimal or equivalent hexadecimal character text.
 
 
Entity reference)
 
 
HTML object, like. When found, sgmlparser uses the HTML object name to call handle_entityref.
 
 
Comment)
 
 
HTML annotation, including in <! Between --... -->. When found, sgmlparser calls handle_comment with the annotation content.
 
 
Processing Instruction)
 
 
HTML processing commands, including. When found, sgmlparser calls handle_pi by processing the instruction content.
 
 
Declaration)
 
HTML declaration, such as doctype, included in <! Between --... -->. When found, sgmlparser uses the declared content to call handle_decl.
 
 
Text Data)
 
 
Text block. It does not satisfy any of the other 7 categories. When found, sgmlparser uses text to call handle_data.
 
 
Important
 
 
Python 2.0 has a bug that sgmlparser cannot recognize declarations at all (handle_decl will never be called ),
 
 
This means that doctype is ignored quietly. This error was corrected in Python 2.1.
 
 
An example of a test suite attached to sgmllib. py illustrates this. You can run sgmllib. py,
 
 
Input the name of an HTML file under the command line, and then print them out while analyzing the tag and other elements.
 
 
Its implementation is to subclass the sgmlparser class, and then define unknown_starttag, unknown_endtag, handle_data
 
 
And other methods. These methods simply print out their parameters.
 
 
For more information about the usage of Python sgmlparser, see Python sgmlparser.
,
 
 
The following is an example I wrote using Python sgmlparsr. I hope it will be helpful to you:
 
 
# Encoding = UTF-8
 
 
# @ Description: baidutiba content
 
 
Import sys
 
 
Import re
 
 
Import httplib
 
Import urllib
 
 
From sgmllib import sgmlparser
 
 
Class baidutiebaparser (sgmlparser ):
 
 
''' The title of the keyword collected in the Baidu Post Bar '''
 
 
Def reset (Self ):
 
 
Sgmlparser. Reset (Self)
 
 
Self.info = [] #
 
 
Self. q_check = 0
 
 
Self. num = 0
 
 
Self. strcontent =''
 
 
 
 
 
Def start_td (self, tag ):
 
 
'''Matching tag '''
 
 
If Len (TAG )! = 0 and tag [0] [1] ='s ':
 
 
Self. num = self. Num + 1
 
 
Self. q_check = 1
 
 
Def handle_data (self, text ):
 
 
'''Processing text '''
 
 
TXT = text. Strip ()
 
 
If TXT and self. q_check:
 
 
For I in checklist:
 
Pipei = r '% s' % STR (I) # Find the matching keyword in the information to be matched
 
 
Check_pan = Re. Compile (pipei)
 
 
If check_pan.search (txt) is not none:
 
 
Self.info. append (txt)
 
 
Else:
 
 
Continue
 
 
Self. strcontent = '$ | $'. Join (self.info)
 
 
Def end_td (Self ):
 
 
'''Matched '''
 
 
Self. q_check = 0
 
 
 
 
 
######################################## #### Configuration information #############################
 
 
Keylist = ['travel '] # post bar name
 
 
Checklist = ['zhangjiajie ', 'korea'] # keywords to be queried
 
 
Content ={}# collected content
 
 
For m in keylist:
 
 
Page = 0
 
 
Keyword = urllib. Quote (M. Decode ('utf-8'). encode ('gbk '))
 
For I in xrange (10 ):
 
 
Url = ''' http: // tieba.baidu.com/F? Z = 0 & Ct = 318767104 & lm = 11 & SC = 0 & Rn = 50 & Tn = baidukeywordsearch & rs3 = 0 & rs4 = 0 & word = % S & Pn = % s ''' % (STR (keyword ), STR (page ))
 
 
Data = urllib. urlopen (URL). Read ()
 
 
Data = Unicode (data, 'gbk'). encode ('utf-8 ')
 
 
Parser = baidutiebaparser ()
 
 
Parser. Feed (data)
 
 
Content [I + 1] = parser. strcontent
 
 
Page = page + 50
 
 
For k in content. Keys ():
 
 
Print K
 
 
Print content [k]
 
 
If you use the python queue Module
If you are interested, you can check it out!
 
 
The above example also uses the python string replacement
.
 
 
Author: Lao Wang @ Python
Python tutorial
 
 
Old Wang Python provides Python-related Python tutorials and Python downloads
Hope you will like it