Automatic extraction algorithm of document Digest--extracting type

Automatic extraction algorithm of document Digest--extracting type

The algorithm of extracting document abstracts automatically, the mainstream method is divided into two categories: extractive extraction, abstractive profile. This article we will mainly extract the formula.

Extraction Type:

Extract some representative pieces of text from the original document set as summaries, which can be sentences, clauses, paragraphs, or subsections throughout the document.

There are two problems with the extraction method, how to rate the order of text units, and how to extract a subset of the text cells to generate a summary. corresponding to the sorting unit and the extraction Unit respectively.

In Layman's words, a sorting unit is used to sort the cells in the document, select the top-ranked units, and then remove the redundant information between the selected units with the extraction unit to get the final automatic summary.

Two implementation ideas: 1 The sorting unit scores the sentence and extracts the subset of the sentences from the unit (redundant) as a summary; 2 The sorting unit scores the concept in the document set, extracting a set of sentences that maximize the important concepts.

There are three kinds of learning sorting algorithms for sorting units:

1) pointwise sort: Treat each sample in isolation (sentence or concept), input the character and mark (rank) of the sentence into the machine learning algorithm, and learn the classifier;

2) pairwise sort: Learn the sort function f (x_i) from a series of sentence pairs or concepts to {(x_i, X_j)}. The sorting problem between 22 samples was considered;

3) Listwise sort: the sort between all samples is taken into account.

To some extent, the methods of learning sort algorithm (LTR) and multi-marker learning are similar, first-order/second-order/advanced methods.

For the extraction unit, we need to extract some representative sentences from the sorted text cells, generate extract summaries, and remove redundancy as much as possible during the extraction process. An extraction method based on integer programming is commonly used .

To solve the above integer programming, we can get the sentences which need to be extracted and composed of abstracts.

Reference documents:

Research and application of document Summarization algorithm Jin Feng

Automatic extraction algorithm of document Digest--extracting type