Vim file encoding and garbled Processing

Reprinted from http://edyfox.codecarver.org/html/vim_fileencodings_detection.html

In vim, there are four encoding-related options: fileencodings, fileencoding, encoding, and termencoding. In actual use, any option error may cause garbled characters. Therefore, each Vim user should clarify the meaning of these four options. The following describes in detail the meanings and functions of these four options.

1 encoding

Encoding is the internal character encoding method used by VIM. After encoding is set, all the buffer, registers, and strings in the script in VIM use this encoding. When Vim is working, if the encoding method is inconsistent with its internal encoding, it will first convert the encoding to the internal encoding. If the encoding used for work contains characters that cannot be converted to internal encoding, these characters will be lost. Therefore, when selecting the vim internal encoding, you must use an encoding with sufficient performance to avoid affecting normal operations.

Since the encoding option involves the internal representation of all characters in Vim, it can only be set once when Vim is started. Modifying encoding in VIM may cause many problems. If there is no special reason, set encoding to UTF-8. To avoid garbled menus and system prompts in non-UTF-8 systems such as Windows, you can also make these settings:

set encoding=utf-8set langmenu=zh_CN.UTF-8language message zh_CN.UTF-8

2 termencoding

Termencoding is the code Vim uses for screen display. During display, VIM converts the internal code to screen encoding and then outputs the code. When the internal encoding contains a character that cannot be converted to screen encoding, the character becomes a question mark, but the editing operation is not affected. If termencoding is not set, encoding is directly used without conversion.

For example, when you log on to the Linux workstation via Telnet in windows, because Windows telnet is GBK encoded, and Linux uses UTF-8 encoding, garbled characters are displayed in VIM in Telnet. At this time, there are two ways to eliminate Garbled text: one is to change Vim's encoding to GBK, the other is to keep Encoding As UTF-8, and change termencoding to GBK, transcode Vim when it is displayed. Obviously, when you use the previous method, if the edited file contains
When GBK cannot represent characters, these characters will be lost. However, if the last method is used, although these characters cannot be displayed due to terminal limitations, they will not be lost during editing.

For gvim in the graphic interface, its display does not depend on the term, so termencoding has no meaning for it. In gvim under gtk2, termencoding is always UTF-8 and cannot be modified. In Windows, gvim ignores the existence of termencoding.

3 fileencoding

When Vim reads a file from a disk, it detects the file encoding. If the file encoding method is different from the vim internal encoding method, VIM converts the encoding method. After the conversion, VIM sets the fileencoding option to the file encoding. If the encoding and fileencoding are different when Vim stores the disk, VIM performs encoding conversion. Therefore, by setting fileencoding after opening the file, we can convert the file from one encoding to another encoding. However, we can see from the previous introduction that fileencoding
It is automatically set after Vim detects a file when it is opened. Therefore, in case of garbled characters, we cannot correct the garbled characters by setting fileencoding again after opening the file.

4 fileencodings

The automatic identification of encoding is implemented by setting fileencodings. Note that it is in the plural form. Fileencodings is a list separated by commas (,). Each item in the list is an encoded name. When we open the file, VIM uses the encoding in fileencodings to try decoding. If it succeeds, it uses this encoding method and sets fileencoding to this value, if the Code fails, test the next encoding.
Therefore, when setting fileencodings, we must put the encoding method that is strictly required and is more prone to decoding failures when the file is not encoded, put the loose encoding method at the end.
For example, Latin1 is a very loose encoding method. The text obtained by any encoding method is decoded using Latin1 and will not fail to be decoded.-Of course, the decoded results are naturally "garbled ". Therefore, if you put Latin1 at the top of fileencodings, opening any Chinese file is garbled.
The following is a fileencodings setting recommended by dianhu:

set fileencodings=ucs-bom,utf-8,cp936,gb18030,big5,euc-jp,euc-kr,latin1

Among them, the ucs-bom is a very strict encoding. files without this encoding are hardly mistaken for the ucs-bom, so they are placed first.

UTF-8 is also quite strict, in addition to very short files (for example, many people relish the GBK encoding of the "Unicom" was misjudged as a classic error of UTF-8 encoding ), in real life, files are almost impossible to be misjudged, so they are placed in the second place.
The following are cp936 and gb18030. These two types of codes are relatively loose. If we put them in front, there will be a lot of misjudgment, So let them back. The encoding space of cp936 is smaller than that of gb18030, so cp936 is placed before gb18030.
As for big5, EUC-JP, and EUC-KR, they are strictly the same as cp936. Put them behind them and there will inevitably be a lot of misjudgment when editing these encoded files, but this is a problem that Vim's built-in encoding detection mechanism cannot solve. Since Chinese users rarely have the opportunity to edit these encoding files, we decided to ensure that cp936 and gb18030 are recognized.
Finally, latin1. It is an extremely loose code, so we have to put it in the last place. Unfortunately, when you encounter a file with Latin1 encoding, in most cases, it does not have the opportunity to fall-back to Latin1, which is often mistaken in the previous encoding. However, as mentioned earlier, Chinese users do not have much access to such files.
If the encoding is wrong, the decoded results won't be recognized by humans, so we can say that this file is garbled. If you know the correct encoding of the file, you can open the file by using ++ ENC = encoding when opening the file, for example:

:e ++enc=utf-8 myfile.txt

5 fencview

According to the previous introduction, we know that the recognition rate is very low through the built-in encoding recognition mechanism of VIM, especially for simplified Chinese (GBK/gb18030) and traditional Chinese (big5) identification Between Japanese (EUC-JP) and Korean (EUC-KR. For common users, it is unrealistic to see the encoding method of a file with the naked eye. Therefore, Dian Hu strongly recommends the fencview plug-in developed by mbbill In the Shui Mu community. This plug-in uses word frequency statistics to identify and encode, with a very high accuracy rate. Click here
Download.