Vim file encoding and garbled Processing

In vim, there are four encoding-related options:Fileencodings,Fileencoding,EncodingAndTermencoding. In actual use, any option error may cause garbled characters. Therefore, each Vim user should clarify the meaning of these four options. The following describes in detail the meanings and functions of these four options.

1 encoding

EncodingIt is the internal character encoding method used by VIM. When we setEncodingAll the buffer, registers, and strings in the script in VIM use this encoding. When Vim is working, if the encoding method is inconsistent with its internal encoding, it will first convert the encoding to the internal encoding. If the encoding used for work contains characters that cannot be converted to internal encoding, these characters will be lost. Therefore, when selecting the vim internal encoding, you must use an encoding with sufficient performance to avoid affecting normal operations.

BecauseEncodingThe option involves the internal representation of all characters in Vim, so it can only be set once when Vim is started. ModifyEncodingThis can cause many problems. If there is no special reason, always setEncodingSetUTF-8. To avoid garbled menus and system prompts in non-UTF-8 systems such as Windows, you can also make these settings:

Set Encoding= UTF-8
Set Langmenu= Zh_CN.UTF-8
LanguageMessage ZH_cn.utf- 8

2 termencoding

termencoding is the code that Vim uses for screen display, during display, VIM converts the internal encoding to screen encoding and then uses it for output. When the internal encoding contains a character that cannot be converted to screen encoding, the character becomes a question mark, but the editing operation is not affected. If termencoding is not set, encoding is used directly without conversion.

for example, when you log on to the Linux workstation via Telnet in windows, because Windows telnet is GBK encoded, and Linux uses UTF-8 encoding, garbled characters are displayed in VIM in Telnet. At this time, there are two ways to eliminate Garbled text: first, use Vim's encoding to GBK , another method is to keep encoding is UTF-8 , set termencoding to GBK enables Vim to transcode when it is displayed. Obviously, when using the previous method, if the edited file contains characters that cannot be expressed by GBK, these characters will be lost. However, if the last method is used, although these characters cannot be displayed due to terminal limitations, they will not be lost during editing.

For gvim in the graphic interface, its display does not depend on the term, soTermencoding It does not make sense. In gvim under gtk2,Termencoding Always UTF-8And cannot be modified. Gvim in Windows ignoresTermencoding .

3 fileencoding

When Vim reads a file from a disk, it detects the file encoding. If the file encoding method is different from the vim internal encoding method, VIM converts the encoding method. After the conversion, VIM willFileencoding Specifies the encoding of the file. IfEncoding AndFileencoding Different, VIM performs encoding conversion. Therefore, after opening the file, SetFileencoding, We can convert the file from one encoding to another encoding. However, we can see from the previous introduction that,Fileencoding It is automatically set after Vim detects a file when it is opened. Therefore, in case of garbled characters, we cannot reset it after opening the file.Fileencoding To correct garbled characters.

4 fileencodings

Encoding is automatically identified by settingFileencodingsIt is implemented in the plural form.FileencodingsIs a list separated by commas (,). Each item in the list is an encoded name. When we open the file, VIM usesFileencodingsIf the encoding method is successful, the encoding method is used for decoding andFileencoding Set it to this value. If it fails, continue to test the next encoding.

Therefore, we are settingFileencodings When the encoding is not the same, it is necessary to put the strict requirements, when the file is not the encoding, it is more likely that the decoding failure encoding method is put in front, put the loose encoding method behind.

For example,Latin1It is a very loose encoding method. The text obtained by any encoding method is decoded using Latin1 and will not be decoded.-Of course, the decoded results are naturally "garbled ". Therefore, if youLatin1 Put itFileencodings In the first place, opening any Chinese file is garbled.

Which of the following isFileencodings Settings:

Set Fileencodings= Ucs-bom ,UTF-8 ,Cp936 ,Gb18030 ,Big5 ,EUC-JP ,EUC-KR ,Latin1

Among them, the ucs-bom is a very strict encoding. files without this encoding are hardly mistaken for the ucs-bom, so they are placed first.

UTF-8 is also quite strict, in addition to very short files (for example, many people relish the GBK encoding of the "Unicom" was misjudged as a classic error of UTF-8 encoding ), in real life, files are almost impossible to be misjudged, so they are placed in the second place.

The following are cp936 and gb18030. These two types of codes are relatively loose. If we put them in front, there will be a lot of misjudgment, So let them back. The encoding space of cp936 is smaller than that of gb18030, so cp936 is placed before gb18030.

As for big5, EUC-JP, and EUC-KR, they are strictly the same as cp936. Put them behind them and there will inevitably be a lot of misjudgment when editing these encoded files, but this is a problem that Vim's built-in encoding detection mechanism cannot solve. Since Chinese users rarely have the opportunity to edit these encoding files, we decided to ensure that cp936 and gb18030 are recognized.

Finally, latin1. It is an extremely loose code, so we have to put it in the last place. Unfortunately, when you encounter a file with Latin1 encoding, in most cases, it does not have the opportunity to fall-back to Latin1, which is often mistaken in the previous encoding. However, as mentioned earlier, Chinese users do not have much access to such files.

If the encoding is wrong, the decoded results won't be recognized by humans, so we can say that this file is garbled. If you know the correct encoding of the file, you can use++ ENC = Encoding To open the file, such:

: E ++ ENC = UTF-8 myfile.txt

5 fencview

According to the previous introduction, we know that the recognition rate is very low through the built-in encoding recognition mechanism of VIM, especially for simplified Chinese (GBK/gb18030) and traditional Chinese (big5) identification Between Japanese (EUC-JP) and Korean (EUC-KR. For common users, it is unrealistic to see the encoding method of a file with the naked eye. Therefore, Shui Mu is strongly recommended by DIAN Hu.CommunityThe fencview plug-in developed by mbbill. This plug-in uses word frequency statistics to identify and encode, with a very high accuracy rate. ClickHereDownload.

 

Note, this article from: http://edyfox.codecarver.org/html/vim_fileencodings_detection.html

 

Generally used:

 
:SetEncoding = UTF-8Termencoding = GBK
 

You can.