Python implementation method of extracting article Abstracts

The example in this paper describes how Python implements extracting abstract articles. Share to everyone for your reference. Specific as follows:

I. Overview

In the blog system article list, in order to more effectively present the content of the article, so that the reader more targeted choice of reading, usually also provides the title and summary of the article.

The content of an article can be in plain text format, but in today's web, more is HTML format. Regardless of the format, the summary is generally the beginning of the article content, you can follow the specified number of words to extract.

Ii. Plain Text Abstracts

A plain text document is a long string, and it is easy to extract a digest of it:

#!/usr/bin/env python#-*-coding:utf-8-*-"" "Get a summary of the Text-format document" "Def get_summary (TEXT, count): 
  u "" Get the first ' count ' characters from ' text '    >>> text = U ' Welcome This is an article about Python '    >>> g Et_summary (text, page) = = U ' Welcome This is a ' True ' ""  assert (isinstance (text, Unicode))  return text[0: Count]if __name__ = = ' __main__ ':  import doctest  doctest.testmod ()

Three, HTML summary

The HTML document contains a large number of markers, such as

、

, and so on), these characters are tag directives and are usually paired, and simple text interception destroys the HTML document structure, which in turn causes the digest to appear improperly in the browser.

When you follow the structure of the HTML document and intercept the content, you need to parse the HTML document. In Python, you can do this with the standard library htmlparser.

One of the simplest abstraction functions is to omit HTML tags and extract only the native text inside the markup. The following is a python implementation similar to this feature:

#!/usr/bin/env python#-*-coding:utf-8-*-"" "Get a raw summary of the Html-format document" "from Htmlparser import HTML Parserclass Summaryhtmlparser (Htmlparser): "" "Parse HTML text to get a summary >>> text = U '
Hi Guys:
This is a example using Summaryhtmlparser.
' >>> parser = Summaryhtmlparser (Ten) >>> parser.feed (text) >>> parser.get_summary (U ' ...    .') U
Higuys:thi ...
' "" "Def __init__ (Self, Count): htmlparser.__init__ (self) self.count = count Self.summary = U ' def feed (self , data): "" "Only accept Unicode ' data '" "" Assert (isinstance (data, Unicode)) htmlparser.feed (self, data) def hand Le_data (self, data): more = Self.count-len (self.summary) If more > 0: # Remove possible whitespaces in ' da Ta ' data_without_whitespace = U '. Join (Data.split ()) Self.summary + = Data_without_whitespace[0:more] def get_su Mmary (Self, suffix=u ", Wrapper=u ' P '): Return U ' <{0}>{1}{2}
  '. Format (wrapper, self.summary, suffix) if __name__ = = ' __main__ ': Import doctest doctest.testmod ()

Htmlparser (or BeautifulSoup, etc.) is more suitable for complex HTML digest extraction, for the above simple HTML Digest extraction function, in fact, there is a more concise implementation of the scheme (compared to summaryhtmlparser):

#!/usr/bin/env python#-*-coding:utf-8-*-"" "Get a raw summary of the Html-format document" "" Import redef get_summary (te XT, Count, Suffix=u ', Wrapper=u ' P '): "" "  A Simpler Implementation (VS ' summaryhtmlparser ').    >>> Text = U '
Hi Guys:
This is a example using Summaryhtmlparser.
'    >>> get_summary (text, ten, U ' ... ')    u '
Higuys:thi ...
' "" "  assert (isinstance (text, Unicode))  summary = re.sub (R ' <.*?> ', U ', text) # key Difference:use Regex  Summary = U '. Join (Summary.split ()) [0:count]  return u ' <{0}>{1}{2}
 
    '. Format (wrapper, Summary, suffix) if __name__ = = ' __main__ ':  import doctest  doctest.testmod ()

Hopefully this article will help you with Python programming.

