Errors Caused by reading UTF-8 encoded text files in a Chinese System

I. Errors Caused by reading UTF-8 encoding files

There is a TXT file with the following content:

aaabbbccc

Open the TXT file in UTF-8 encoding mode, read in sequence, put the value in a hashset, and determine whether AAA is in hashset

Class {public static void main (string [] ARGs) {try {hashset <string> specialcateset = new hashset <string> (); fileinputstream A = new fileinputstream ("D: // test.txt "); string encoding =" UTF-8 "; inputstreamreader B = new inputstreamreader (A, encoding); bufferedreader BR = new bufferedreader (B); string message = BR. readline (); While (message! = NULL) {specialcateset. add (Message); message = BR. readline () ;}} catch (exception e) {e. printstacktrace ();} If (specialcateset. contains ("AAA") {system. out. println ("specialcate contains AAA");} else {system. out. println ("specialcate does not contain AAA ");}}}

Result output: specialcateset does not contain aaa

The value in the resumable trace specialcateset does include the value "AAA", but why does specialcateset. Contains ("AAA") verify that it is false?

Ii. Cause of error

In Chinese operating system, for text files in UTF-8 encoding format, the value of the first three bytes is-17,-69,-65 (these three bytes illustrate the encoding method, for more information, see How to obtain the file encoding format in Java)

The modification procedure is as follows:

Class {public static void main (string [] ARGs) {try {hashset <string> specialcateset = new hashset <string> (); fileinputstream A = new fileinputstream ("D: // test.txt "); string encoding =" UTF-8 "; inputstreamreader B = new inputstreamreader (A, encoding); bufferedreader BR = new bufferedreader (B); string message = BR. readline (); While (message! = NULL) {// print the byte value of the Output Message. byte [] a1 = message. getbytes (); For (byte B1: A1) {system. out. print (b1 + "");}
System. Out. println (); specialcateset. Add (Message); message = Br. Readline () ;}} catch (exception e) {e. printstacktrace ();}}}

The output result is as follows, where ascll of A is 97, B: 98, C: 99.
-17 69 65 97 97

98 98 98

99 99 99

Therefore, specialcateset stores three string values AAA (byte:-17 69 65 97 97), BBB (byte:-98 98 ), CCC (byte:-99 99). Although specialcateset in Java contains the string AAA, however, the byte of the string AAA is (-17 69 65 97 97 97), and specialcateset is determined later. contains ("AAA"), the AAA byte defined in Java is (97 97), so the verification fails.

In general, the AAA byte read in a text file is (-17 69 65 97 97), which contains three bytes of the encoding value, however, when the code is displayed in Java, the encoding value of the first three digits is ignored. Therefore, the value displayed in Java is AAA, the Byte byte of the string AAA defined in Java is (97 97 97). Although the two are consistent, their values are different and Their hashcode values are different, therefore, Java judges that this AAA is different from the AAA

 

Iii. Solution

1. the simplest one is to leave the first line in the TXT blank, because the first three digits are the encoding information.

2. read the first line and remove the encoding information.

 

Errors Caused by reading UTF-8 encoded text files in a Chinese System