Special case of UTF-8 coding failure

The method for detecting UTF-8 encoding in ruby is the string # unpack method, passed in the "u *" template. This template means to regard the string as a UTF-8 string, each UNICODE character is converted to the corresponding Code Points are combined into arrays. The string itself is indeed a UTF-8, it can be converted successfully, otherwise an exception will be thrown. According to this behavior, add the following instance method to string:

Class string
Def utf8?
Unpack ('U * ') Rescue return false
True
End
End

AboveProgramIn most cases, it is true. But I found that in gb2312 encoding, The gb2312 encoding of the following Chinese characters can pass the above method of detection, and was mistaken for a UTF-8 string:

Bit, front, branch, school, write, Yuan

There may be some more. Therefore, the above method is not suitable for UTF-8 encoding detection of Chinese strings that may contain gb2312 encoding.

Note that this is not a ruby problem, but an essential problem caused by different encoding modes.

You can do the following experiments:
 
1. Open notepad and write a 'bit'. Save it in ANSI encoding without pressing the Enter key (note that it must be saved in ANSI encoding ).

2. open the file again and you will see that it is not a "bit", but a lambda.

Similarly, there are similar phenomena in words such as "front, branch, school, write, and Yuan.

It is not easy to detect string encoding. I wonder if there are any reliable methods (except for direct trial transfer using the iconv library.