(Original) a worrying garbled troubleshooting process

In many cases, the problem is that you are too superficial.

A function is required in the project. A robot simulates and chats with humans. After a player speaks a sentence, the robot instinctively starts chatting with him, I think that as long as there is a powerful dictionary and sharding algorithm, it's just a bit of something, but it's really a lot of pressure to do it on your own. So I searched the internet decisively and found this thing easily:

 

This gives me the first feeling that it is real, yes, and can fully meet the needs, but it seems that it does not provide an interface. This is not a problem. I will definitely flip the webpage source code and find the post place, it is an ajax post request with a parameter, which is very simple, so I tried to simulate HTTP in java in minutes and wrote a junit test:

 

In just 10 minutes, an intelligent chatbot was able to solve the problem, so it was easy to deploy it to jetty. Many people have heard about jetty, the EOF in websocket has been used for a long time. After the deployment, the client starts to use the Android client for testing. The client sends one: Hello, and the other client receives the following message:What kind of tea is there?

It is obviously garbled. I feel like debugging:

 

We can see that the returned http value has been garbled. In this process, we have gone through the following steps:

Send http request ---> third-party interface http return value --> Local jetty container accepts binary stream --> jetty recodes binary data based on container encoding --> Java InputStream receives the stream after jetty encoding --> java convert it to String

 

So we can see the return value of a String, which is garbled. Generally, there are two types of Chinese characters garbled:

1:What kind of tea is there?In fact, such garbled characters are not called garbled characters, but data is not what we want, because what we want is A but B. This is mainly because the encoding format is incorrect.

2:?????The garbled characters with all question marks should have been met by many people. Such things should be garbled. Why ?. Because something in a byte cannot be displayed with a Chinese character, that is, the content corresponding to the Chinese character cannot be found, so such a thing will take? The official name is the encoding black hole. The corresponding binary data is 63. After conversion, it is?

 

According to the situation, what I encountered was the first one, so I was a bit confused. I decided to turn it into a strong one:

ChangeCharset changeCharset = new ChangeCharset (); try {result = changeCharset. toUTF_8 (URLDecoder. decode (result, "UTF-8");} catch (UnsupportedEncodingException e) {e. printStackTrace ();} return result;

What encoding do you care about, brother gives you a unified, so the United States once again open the client test, there is another situation:

 

Robot said: Hello, love you, dear ??

I tried to find out some garbled characters, so I continued the test to find out the pattern. Later, I found that as long as there are even numbers of data, there will be no garbled characters. If there are odd numbers, then the last Chinese character is garbled. Is the garbled form fixed ?, Here is one ?, I rely on it. Today we have met both situations. I thought that something very simple was stuck in the coding place and thought hard. It was obviously a container encoding problem, A very SB solution is also very simple. You just need to judge whether it is even or not, rather than adding something even.

 

But want to understand why is the last word garbled, suddenly think of a thing, UTF-8, a Chinese character 3 bytes, GBK a Chinese character 2 bytes, I seem to understand what ..

 

Because jetty containers determine the container encoding according to the system encoding by default, the premise is that they do not modify the startup encoding themselves, and in the company, My PC is windows, as if the default is GBK, anyway, I have a lot of windows gossip, so there is a problem here, for example jetty received a string of UTF-8-encoded Chinese characters:

I'm fine.

The most primitive binary array received by jetty is as follows:

[-26,-120,-111,-27,-66,-120,-27,-91,-67]

Of course, this is not the most primitive. The most primitive values are 0 and 1. Of course, for good looks, even if it is the most primitive, jetty will start encoding according to jetty's GBK encoding, it is encoded in two bytes and one Chinese character format, so such a combination occurs:

[-26,-120] [-111,-27] [-66,-120] [-27,-91] [-67]

The corresponding Chinese characters can be found in each of the preceding two bytes. jetty finds that there is only one byte in the end and cannot find the corresponding Chinese character. In my mind, jetty gave up on it, drive it out and drop 63, so the final combination is:

[-26,-120] [-111,-27] [-66,-120] [-27,-91] [63]

After encoding in GBK format, the two bytes correspond to one Chinese character, and such a thing is displayed:

Braised tea?

Five characters are displayed, because each two bytes represents a Chinese character, the last byte is 63, and the corresponding symbol is ?, The above thing appeared, so I made a forced UTF-8 code on it, resulting in the above binary array re-combination, after the combination of the UTF-8, the binary array became like this:

[-26,-120,-111] [-27,-66,-120] [-27,-91, 63]

After the UTF-8 display, it becomes like this:

I am very ??

The first six bytes can display Chinese characters normally, because they are real data. However, the last three bytes have been processed and replaced by GBK, even if you use a UTF-8, it cannot restore its original appearance, so it is displayed as above, but why not even errors?

Because even numbers can be decoded by GBK, that is, if the Chinese character is even, UTF-8 and GBK is equivalent, but if it is odd, then the problem, this is also the legend of the last Chinese character garbled problem, because the last byte is always 63. To solve this problem, it is necessary to cure the problem and cure the problem. The project must ensure the encoding consistency throughout the process, because my project is a game server, WebSocket is used. If Servlet is used, it can be processed directly in Servlet or Filter.

Different encoding methods have different processing logic. Many times, we seem to have solved the problem, but the problem is not exposed. Knowing its root cause, we can strategize and win a thousand miles away.

 

I don't know what I took in the middle of the night a few days ago. I upgraded Centos to 6.6. As a result, all eclipse cannot be opened under 6.6, which is a fatal-level error, it was not something that I could solve by other people. The update package downloaded during the update was 1.4 GB, which is obviously a problem of system compatibility, so I thought that a code farmer could not die in eclipse, so I threw down VIM and got several plug-ins. Although VIM was used in the past, but it was not so formal this time, it was found that this stuff is actually quite good, basically except for the breakpoint Debug of eclipse, the other is similar to him, writing C/C ++ is faster. It is quite convenient for a \ im to annotate the entire file as a whole, and the UI under Linux is quite good, especially for DUCK desktop, similar to Mac, you can even write something like this in java:

For programmers, Linux is a good thing. If you have friends who like to test the water, you can try it.

Attach the beauty photos that have recently added DUCK to your system:

 

Conclusion) in fact, this problem is very simple, but it was too SB at the time, so I am worried about it ..

(Original) a worrying garbled troubleshooting process