Vim file encoding and garbled Processing

In vim, there are four encoding-related options:fileencodings,fileencoding,encodingAndtermencoding. In actual use, any option error may cause garbled characters. Therefore, each
Vim users should clarify the meaning of these four options. The following describes in detail the meanings and functions of these four options.

1 encoding

encodingIt is the internal character encoding method used by VIM. When we setencodingAfterwards, VIM
All the buffer, registers, and strings in the script use this encoding. When Vim is working, if the encoding method is inconsistent with its internal encoding, it will first convert the encoding to the internal encoding. If the encoding used for work contains characters that cannot be converted to internal encoding, these characters will be lost. Therefore, when selecting the vim internal encoding, you must use an encoding with sufficient performance to avoid affecting normal operations.

BecauseencodingThe option involves the internal representation of all characters in Vim, so it can only be set once when Vim is started. ModifyencodingThis can cause many problems. If there is no special reason, always setencodingSetutf-8. To avoid
UTF-8 systems such as Windows, menus and system prompts appear garbled, You can do these settings at the same time:

Set encoding = UTF-8
Set langmenu = zh_CN.UTF-8
Language message zh_CN.UTF-82 termencoding

termencodingVim is the code used for screen display. When it is displayed, VIM converts the internal code to screen encoding and then uses it for output. When the internal encoding contains a character that cannot be converted to screen encoding, the character becomes a question mark, but the editing operation is not affected. IftermencodingIf not set, useencodingNo conversion is performed.

For example, when you log on to the Linux workstation via Telnet in windows, because Windows telnet is GBK encoded, and Linux uses UTF-8 encoding, garbled characters are displayed in VIM in Telnet. At this time, there are two ways to eliminate Garbled text: first, use Vim'sencodingChangegbk, Another way is to keepencodingIsutf-8termencodingChangegbk, Let
Transcoding when Vim is displayed. Obviously, when using the previous method, if the edited file contains characters that cannot be expressed by GBK, these characters will be lost. However, if the last method is used, although these characters cannot be displayed due to terminal limitations, they will not be lost during editing.

For gvim in the graphic interface, its display does not depend on the term, sotermencodingIt does not make sense. In gvim under gtk2,termencodingAlwaysutf-8And cannot be modified. While
Gvim in Windows is ignored.termencoding.

3 fileencoding

When Vim reads a file from a disk, it detects the file encoding. If the file encoding method is different from the vim internal encoding method, VIM converts the encoding method. After the conversion, VIM willfileencodingSpecifies the encoding of the file. When Vim
When saving the disk, ifencodingAndfileencodingDifferent, VIM
It will undergo encoding conversion. Therefore, after opening the file, Setfileencoding, We can convert the file from one encoding to another encoding. However, we can see from the previous introduction that,fileencodingWhen opening a file
Vim is automatically set after testing. Therefore, in case of garbled characters, we cannot reset it after opening the file.fileencodingTo correct garbled characters.

4 fileencodings

The automatic identification of encoding is implemented by setting fileencodings. Note that it is in the plural form. Fileencodings is a list separated by commas (,). Each item in the list is an encoded name. When we open the file, VIM uses the encoding in fileencodings in sequence to try decoding. If it succeeds, it uses this encoding method andfileencodingSet it to this value. If it fails, continue to test the next encoding.

Therefore, we are settingfileencodingsWhen the encoding is not the same, it is necessary to put the strict requirements, when the file is not the encoding, it is more likely that the decoding failure encoding method is put in front, put the loose encoding method behind.

For example, Latin1 is a very loose encoding method. The text obtained by any encoding method is decoded using Latin1 and will not fail to be decoded.-Of course, the decoded results are naturally "garbled ". Therefore, if youlatin1Put itfileencodingsIn the first place, opening any Chinese file is garbled.

Which of the following isfileencodingsSettings:

Set fileencodings = ucs-bom, UTF-8, cp936, gb18030, big5, EUC-JP, EUC-KR, Latin1

Among them, the ucs-bom is a very strict encoding. files without this encoding are hardly mistaken for the ucs-bom, so they are placed first.

UTF-8 is also quite strict, in addition to very short files (for example, many people relish the GBK encoding of the "Unicom" was misjudged as a classic error of UTF-8 encoding ), in real life, files are almost impossible to be misjudged, so they are placed in the second place.

The following are cp936 and gb18030. These two types of codes are relatively loose. If we put them in front, there will be a lot of misjudgment, So let them back. The encoding space of cp936 is smaller than that of gb18030, so cp936 is placed before gb18030.

As for big5, EUC-JP, and EUC-KR, they are strictly the same as cp936. Put them behind them and there will inevitably be a lot of misjudgment when editing these encoded files, but this is a problem that Vim's built-in encoding detection mechanism cannot solve. Since Chinese users rarely have the opportunity to edit these encoding files, we decided to ensure that cp936 and gb18030 are recognized.

Finally, latin1. It is an extremely loose code, so we have to put it in the last place. Unfortunately, when you encounter a file with Latin1 encoding, in most cases, it does not have the opportunity to fall-back to Latin1, which is often mistaken in the previous encoding. However, as mentioned earlier, Chinese users do not have much access to such files.

If the encoding is wrong, the decoded results won't be recognized by humans, so we can say that this file is garbled. If you know the correct encoding of the file, you can use++enc=encodingTo open the file, such:

:e ++enc=utf-8 myfile.txt

5 fencview

According to the previous introduction, we know that the recognition rate is very low through the built-in encoding recognition mechanism of VIM, especially for simplified Chinese (GBK/gb18030) and traditional Chinese (big5) identification Between Japanese (EUC-JP) and Korean (EUC-KR. For common users, it is unrealistic to see the encoding method of a file with the naked eye. Therefore, Dian Hu strongly recommends the fencview plug-in developed by mbbill In the Shui Mu community. This plug-in uses word frequency statistics to identify and encode, with a very high accuracy rate. Click here to download.

Original article: Vim file encoding and garbled Processing

Http://edyfox.codecarver.org/html/vim_fileencodings_detection.html