Summary of SMT steps

This article is summarized in the series of tutorials in the niutrans forum.

1. Corpus preprocessing

The pre-processing result is a file generated after the bilingual word segmentation. In this step, you must note that the regular phrases, such as numbers, dates, and URLs, are generalized. You can use the regular expression or other methods. Note that vertices in dates and vertices in foreign names and URLs must be separated from vertices at the end of a sentence, and numbers and dates must also be separated.

The change is relatively large and difficult to handle. It should be a Chinese date, because its format is quite variable. One way to extract Chinese Date and time from text (or further convert it to a standard time format) is to use the trie tree for extraction, compared with writing a lot of messy regular expressions, it has the advantages of clear logic, simple extension, and low Code complexity. The specific algorithm process will be detailed in subsequent articles. As far as SMT functions are concerned, in this step, we only need to generalize the time formats with a slight rule, such as 2014-11-: 15; for example, [half past six P.M., January 1, March 8] [, the next day], it is unnecessary to generalize the data. The number contained in the data can be generalized without affecting the accuracy of the results.

In the bilingual after generalization (especially in different languages, such as Chinese and English), it is best to check the generalization consistency, such as the number of numbers in Chinese and the corresponding number in English, the number of time and date URLs must be equal.

The syntax and semantics of the ancient Chinese text are different from those of the current Chinese text. This step can be weakened or even avoided. That is to say, you only need simple word segmentation. The word segmentation method of Ancient Chinese text also needs to be considered. In practice, we adopt one-dimensional word segmentation, that is, one word.

2. Word Alignment

The word alignment phase uses Giza ++ for unidirectional alignment (implementing IBM Model 1 ~ 5. I am still learning the specific algorithm), and then combine the two unidirectional alignment results to generate the final alignment results using the symmetric algorithm.

The symmetric algorithm is like this. First, a matrix is generated based on the two results. Mij saves alignment results: 1) I and j are bidirectional alignment; 2) I alignment J; 3) J alignment I; 4) I and j are not alignment.

1. Add all 1) nodes to the result set;

2. Check the Left and Right neighbors of each node in the result set. There are isolated and unmaster neighbors (that is, only two or three adjacent to the result set), and they are absorbed in the past;

3, the remaining isolated 2) or 3) are also added to the result set.

Complete. The image will not be painted.

Summary of SMT steps