A little exploration of the problem of solving garbled characters (involving Unicode (utf-16) and GBK)

In the previous log (link), we discussed the Utf-8 encoding and GBK encoding conversion between the garbled problem, this article we discuss the Unicode (UTF-16 encoding) and GBK encoding conversion between the garbled problem.

In the notepad that comes with the Windows system, we save using Unicode encoding as shown in the figure.

In Visual Studio 2005, click File | Advanced save options to select unicode-code page 1200.

Only garbled and ASCII code in the file

According to the method in the previous log, we use the Winhex software to view the file's 16 binary data, as follows:

Among them, the first two bytes "FF FE" represents utf-16 BOM code, starting from D4 is the data, namely:

0xd4    0x00    0xDA    0x00    0xb4    0x00    0XCB    0x00    0XCC    0x00    0xed    0x00    0XBC    0x00    0xd3    0x00    0xd7    0x00    0xa8    0x00    0xd3    0x00    0xc3    0x00    0xb4    0x00    0XFA    0x00    0xc2    0x00    0xeb    0x00    0xba    0x00    0xCD    0x00    0x2f    0x00    0XBB    0x00    0xf2    0x00    0xb5    0x00    0xf7    0x00    0xd3    0x00    0xc3    0x00    0XBB    0x00    0xf9    0x00    0xc0    0x00    0xe0    0x00

We also find the GBK encoded value of the correct character

0xd4    0xDA    0xb4    0XCB    0XCC    0xed    0XBC    0xd3    0xd7    0xa8    0xd3    0xc3    0xb4    0XFA    0xc2    0xeb    0xba    0xCD    0x2f    0XBB    0xf2    0xb5    0xf7    0xd3    0xc3    0XBB    0xf9    0xc0    0xe0

It is easy to see that the number of 0x00 in an even sequence is eliminated, and of course our format is changed from Utf-8 to GBK.

The corresponding C language program is as follows:

#include <stdio.h>#include<stdlib.h>intMainintargcChar Const*argv[]) {FILE*FP; FILE*FP2; //Open stored garbled file, utf-16 encoded format, binary open    if((Fp2=fopen ("BadCode.txt","rb+"))==NULL) {printf ("Open Source File failed! \ n"); System ("Pause"); Exit (1); }    //Open, new file to store processed data    if((Fp=fopen ("BadCodeH.txt","w+"))==NULL) {printf ("open/create Destination File failed!\n"); System ("Pause"); Exit (1); }    //garbled first byte of highunsigned ch; //garbled first Low byteunsigned cl; //garbled second High byteunsigned CH2; //garbled second low byteunsigned Cl2; CH=fgetc (FP2); //Discard BOM byte information    if(ch==0xFF) {fgetc (FP2); CH=fgetc (FP2); } CL=fgetc (FP2);  while(!feof (FP2)) {        //garbled, then read two bytes, and then discard even the number section        if(ch>0x7f&& cl==0x00) {CH2=fgetc (FP2); Cl2=fgetc (FP2);            FPUTC (CH,FP);        FPUTC (CH2,FP); }        //ASCII code, discard even digit section        Else{FPUTC (CH,FP); } CH=fgetc (FP2); CL=fgetc (FP2);    } fclose (FP);    Fclose (FP2); System ("Pause"); return 0;}

The results of the operation are as follows:

A more general case (there are normal Chinese characters, garbled characters, and ASCII character in the file)

As in the previous log, for the normal utf-16 encoded characters, we just convert it to GBK encoded output, it is important to note that the normal utf-16 character encoding is stored in the file: The high byte holds the last two bits of the code, and the low byte holds the first two digits of the encoding.

C language Program as follows, stamp here to download the UnicodeToGBK.txt file:

#include <stdio.h>#include<stdlib.h>//reading data from the utf-16 and GBK conversion tablesBOOLReadmap (unsigned*mapvalue) {    //declaring file Pointersfile*FP; //to open a text file of mapped data in a read-write manner    if(NULL = = (fp = fopen ("UnicodeToGBK.txt","r+")) {printf ("error!"); System ("Pause"); return false; }        //A string that stores the Unicode 16 binary data    Charutfstr[4]; //A string that stores the GBK 16 binary data    Chargbkstr[4]; //Storage of Unicode16 binary dataunsigned utfid; //Storage of GBK 16 binary dataunsigned gbkid; //temporary variables for handling characters    CharC; //reading Data     while(!feof (FP)) {        //a string that reads a Unicode valueFread (UTFSTR,4,1, FP); //convert to integral typeSSCANF (UTFSTR,"%4x",&Utfid);        FGETC (FP); //a string that reads the GBK valueFread (GBKSTR,4,1, FP); //convert to integral typeSSCANF (GBKSTR,"%4x",&gbkid);        FGETC (FP); //Assign Valuemapvalue[utfid]=Gbkid;    } fclose (FP); return true;}intMainintargcChar Const*argv[]) {FILE*FP; FILE*FP2; //Open stored garbled file, utf-16 encoded format, binary open    if((Fp2=fopen ("BadCode.txt","rb+"))==NULL) {printf ("Open Source File failed! \ n"); System ("Pause"); Exit (1); }    //Open, new file to store processed data    if((Fp=fopen ("BadCodeH.txt","w+"))==NULL) {printf ("open/create Destination File failed!\n"); System ("Pause"); Exit (1); } unsigned mapvalue[65536]; if(!Readmap (Mapvalue)) {printf ("Convert failed!\n"); System ("Pause"); Exit (1); }    //garbled first byte of highunsigned ch; //garbled first Low byteunsigned cl; //garbled second High byteunsigned CH2; //garbled second low byteunsigned Cl2; //store UTF-16 encoded valuesunsigned utf; //store GBK encoded valuesunsigned GBK; CH=fgetc (FP2); //Discard BOM byte information    if(ch==0xFF) {fgetc (FP2); CH=fgetc (FP2); } CL=fgetc (FP2);  while(!feof (FP2)) {        //garbled, then read two bytes, and then discard even the number section        if(ch>0x7f&& cl==0x00) {CH2=fgetc (FP2); Cl2=fgetc (FP2);            FPUTC (CH,FP);        FPUTC (CH2,FP); }        //ASCII code, discard even digit section        Else if(ch<=0x7f&& cl==0x00) {FPUTC (CH,FP); }        //Otherwise, the normal character, the output after encoding conversion        Else{UTF=cl* the+ch; GBK=Mapvalue[utf]; FPUTC (GBK/ the, FP); FPUTC (GBK% the, FP); } CH=fgetc (FP2); CL=fgetc (FP2);    } fclose (FP);    Fclose (FP2); System ("Pause"); return 0;}

Examples are as follows:

In the future I will be the two articles in the integration of the program, write a graphical interface of the program out.

If there is a mistake, I implore you to teach the Hero ~ ~

A little exploration of the problem of solving garbled characters (involving Unicode (utf-16) and GBK)