Differences between character sets such as ANSI, Unicode, UTF-8, and DBCS, and related data types and functions

Unicode Environment Settings
When installing Visual Studio, you must add the Unicode option when selecting VC ++ to ensure that the relevant library files can be copied to system32.

Unicode compilation settings:
C/C ++, Preprocessor difinitions remove _ MBCS, add _ Unicode, Unicode
Set entry to wwinmaincrtstartup in projectsetting/link/output.
Otherwise, it is compiled for MBCS (ANSI.

UNICODE: Wide-Byte Character Set

1. How to obtain the number of characters in a string that contains both single-byte and double-byte characters?
You can call the Runtime Library of Microsoft Visual C ++ to contain the function _ mbslen to operate multi-byte strings (including single-byte and dual-byte strings.
Calling the strlen function does not really know how many characters are in the string. It only tells you how many bytes are before the end of 0.
Size_t strlen (const char * string );
Size_t wcslen (const wchar_t * string );
Size_t _ mbslen (const unsigned char * string );
Size_t _ mbstrlen (const char * string );

2. How to operate on DBCS strings?
Function Description
Ptstr charnext (lpctstr); returns the address of the next character in the string
Ptstr charprev (lpctstr, lpctstr); returns the address of the previous character in the string.
Bool isdbcsleadbyte (byte); if this byte is the first byte of the DBCS character, a non-zero value is returned.

3. Why Unicode?
(1) It is easy to exchange data between different languages.
(2) enable you to allocate a single. EXE file or DLL file that supports all languages.
(3) improve the running efficiency of applications.
Windows 2000 is developed from scratch using Unicode. If you call any windows function and pass it an ANSI string, the system must first convert the string to Unicode, then, the Unicode string is passed to the operating system. If you want the function to return an ANSI string, the system first converts the Unicode string to an ANSI string and then returns the result to your application. To convert these strings, the system time and memory are required. By developing applications with Unicode from the beginning, you can make your applications run more effectively.

Windows CE itself is an operating system that uses Unicode and does not support ANSI Windows functions.
Windows 98 only supports ANSI and can only develop applications for ANSI.
When Microsoft converts com from a 16-bit windows to Win32, the company determines that all the COM interface methods that require strings can only accept Unicode strings.

4. How to compile Unicode source code?
Microsoft has designed windowsapi for Unicode to minimize the impact of code. In fact, you can write a single source code file to compile it with or without Unicode. You only need to define two macros (Unicode and _ Unicode) to modify and re-compile the source file.

_ Unicode macro is used for the C Runtime header file, while Unicode macro is used for the Windows header file. When compiling the source code module, these two macros must be defined at the same time.

5. What Unicode data types are defined in windows?
Data Type description
Wchar Unicode Character
Pwstr pointer to Unicode string
Pcwstr pointer to a constant Unicode string
The corresponding ANSI data types are char, lpstr, and lpcstr.
The Common Data Types of ANSI/Unicode are tchar, ptstr, and lpctstr.

6. How to operate Unicode?
Character Set feature instance
ANSI operation functions start with str strcpy
Unicode operation functions start with the WCS wcscpy
The MBCS operation function starts with _ MBS _ mbscpy
ANSI/Unicode operation functions start with _ TCS _ tcscpy (C Runtime Library)
ANSI/Unicode operation functions start with lstr lstrcpy (Windows function)
All new and outdated functions have both ANSI and Unicode versions in Windows2000. Functions of the ANSI version end with a, and functions of the Unicode version end with W. Windows will be defined as follows:

# Ifdef Unicode
# Define createmediawex createmediawexw
# Else
# Define createmediawex createmediawexa
# Endif //! Unicode

7. How do I represent Unicode string constants?
Character Set instance
ANSI "string"
Unicode L "string"
ANSI/Unicode T ("string") or _ text ("string ")
If (szerror [0] ==_text ('J ')){}

8. Why should I try to use operating system functions?
Secret. Because these functions are used a lot, they may have been loaded into RAM when the application is running.

Such as strcat, strchr, strcmp, and strcpy.

9. How do I write ANSI and Unicode-compliant applications?
(1) treat a text string as a character array instead of a chars array or byte array.
(2) Use common data types (such as tchar and ptstr) for text characters and strings.
(3) Use explicit data types (such as byte and pbyte) for byte, byte pointer, and data cache.
(4) use the text macro for the original characters and strings.
(5) perform global replacement (for example, replace pstr with ptstr ).
(6) Modifying string operations. For example, a function usually needs to pass a cached size in characters, rather than bytes. This means that sizeof (szbuffer) should not be passed, but sizeof (szbuffer)/sizeof (tchar) should be passed ). In addition, if you need to allocate a memory block to the string and have the number of characters in the string, remember to allocate memory by byte. That is to say, you should call

Malloc (ncharacters * sizeof (tchar) instead of calling malloc (ncharacters ).

10. How to compare the selected strings?
It is implemented by calling comparestring.
Int comparestring (
Lcid locale, // locale identifier
DWORD dwcmpflags, // comparison-style options
Lptstr lpstring1, // pointer to first string
Int cchcount1, // size, in bytes or characters, of first string
Lptstr lpstring2, // pointer to second string
Int cchcount2 // size, in bytes or characters, of second string
);
Locale local comparison definition
Locale_user_default
Locale_system_default

Logo meaning
Norm_ignorecase ignores uppercase and lowercase letters
Norm_ignorekanatype does not distinguish hirakana from katakana
Norm_ignorenonspace ignore no delimiter
Norm_ignoresymbols ignore symbols
Norm_ignorewidth does not distinguish between single-byte characters and double-byte characters.
Sort_stringsort uses punctuation marks as common symbols.

11. How can I determine whether a text file is ANSI or Unicode?
If the first two bytes of a text file are 0xff and 0xfe, Unicode is used; otherwise, ANSI is used.

12. How can I determine whether a string is ANSI or Unicode?
Use istextunicode for determination. Istextunicode uses a series of statistical and qualitative methods to guess the cached content. Because this is not an exact scientific method, istextunicode may return incorrect results.

13. How to convert a string between Unicode and ANSI?
The Windows function multibytetowidechar is used to convert a multi-byte string to a wide string. The function widechartomultibyte converts a wide string to an equivalent multi-byte string.

14. Differences between Unicode and DBCS
Unicode (especially in the C programming language environment) "wide character set ". 「 Every character in Unicode is a 16-Bit Width, not an 8-Bit Width .」 In Unicode, there is no meaning to simply use an 8-bit value. In contrast, we still PROCESS 8-bit values in the double-bit character set. Some bit groups Define characters, while some bit groups show that one character needs to be defined together with another bit group.

Processing DBCS strings is messy, but processing Unicode text is like processing ordered text. You may be glad to know that the first 128 Unicode characters (16-bit code from 0x0000 to 0x007f) are ASCII characters, and the next 128 Unicode characters (code from 0x0080 to 0x00ff) is an ASCII extension of ISO 8859-1. The characters in different parts of Unicode are also based on the existing standards. This is to facilitate conversion. The Greek alphabet uses code from 0x0370 to 0x03ff, the codes from 0x0400 to 0x04ff are used in the Slavic language, the codes from 0x0530 to 0x058f are used in the United States, and the codes from 0x0590 to 0x05ff are used in the Hebrew language. Hieroglyphics (CJK) in China, Japan, and South Korea occupy code from 0x3000 to 0x9fff. The biggest advantage of Unicode is that there is only one character set and there is no ambiguity.

15. Derivative criteria
Unicode is a standard. UTF-8 is its conceptual subset, and UTF-8 is the specific coding standard. Unicode is the standard for all requests to meet the world's unified encoding standards. The UTF-8 standard is a form of deformation of Unicode (iso000046) standard,

UTF is: Unicode/UCOS Transformation format, in fact there are two UTF, one is the UTF-8, one is the UTF-16,
But the UTF-16 is used less, its correspondence is as follows:
In Unicode, the encoding format is 0 xxxxxxx in the UTF-8 encoded as 0000-007f
In Unicode, the encoding format is 0080 XXXXX 10 xxxxxx in the UTF-8 encoded as 110-07ff
In Unicode, the encoding format is: 0000 XXXX 10 xxxxxx 10 xxxxxx in the UTF-8 encoded as 1110-007f

It is a new Unicode encoding standard. In fact, there are several Unicode standards. we know that the Unicode Character inner code used for a long time is 16 bits. In fact, it cannot compile all the characters in the world in a flat system, such as Chinese Tibetan and other small languages, therefore, UTF-8 is extended to 32 bits. That is to say, in theory, UTF-8 can contain the 2nd power character. unicode is designed to encode all characters in a unified way. big5 and GB are independent character sets. This is also called the Far East character set. If you get it to the German version of Windows, it may cause character encoding conflicts .... in earlier windows, the default character set was ANSI. the Chinese characters entered in Notepad are locally encoded, But Unicode can be directly supported within NT/2000. Notepad.exe is an ANSI character in Win95 and 98, and Unicode. ANSI and Unicode are used in NT to conveniently implement corresponding ing, that is, conversion.
ASCII is a character set within the 8-bit range. It cannot be expressed for characters out of the range, such as Chinese characters. Unicode is a character set within the 16-bit range. For character Partition Distribution in different regions, Unicode is a character encoding standard jointly developed by multiple it giants. In Unicode environments such as Windows NT, a single character occupies 16 bytes, while in ANSI environments such as Windows 98, the next character occupies 8 bytes. the Unicode character is a 16-bit width, which can contain a maximum of 65,535 characters. The data type is called wchar.

For existing ANSI characters, Unicode simply expands to 16 bits: for example, ANSI "A" = 0x43, the corresponding Unicode is
"A" = 0x0043
ASCII is used to store 128 characters in seven days. ASCII is a real American standard, so it cannot meet the needs of other countries. For example, the characters and characters in the Slavic language appear in the Windows ANSI character set, it is an extended ASCII code. It stores 8 characters, and the original ASCII code is still stored at a low of 128 characters,

The 128 characters in height are added with Greek letters.
If def Unicode
Tchar = wchar
Else
Tchar = char
You need to add Unicode and _ Unicode to project/settings/C ++/preprocesser definitions.
Uincode, _ Unicode must be defined. If _ Unicode is not defined, settext (hwnd, lpctstr) will be interpreted as settexta (hwnd, lptstr). In this case, the API regards the Unicode string you give as an ANSI string, garbled characters are displayed. Because Windows APIs are compiled and stored in DLL, both Unicode and ANSI strings are considered as a buffer, for example, "0b A3 00 35 24 3C 00 00" if read by ANSI, because the ANSI string ends with '/0', only two bytes can be read. "0b A3/0 ", if you read data by Unicode, the complete read '/123456' ends.

Since Unicode does not have any additional indications, the system must know the format of the string you provide. In addition, Unicode seems to be specified by ansi c ++, and _ Unicode is provided by the Windows SDK. If you do not write a Windows program, you can only define Unicode.

Development Process:
It is centered on file read/write and string processing. There are two main types of files:. txt and. ini files.
In Unicode and non-Unicode environments, strings are processed differently. You need to refer to the preceding two articles (9 and 10) to meet the character string processing requirements in different environments.
The same is true for reading and writing files. When you call related interface functions, add _ text and other related macros to the strings in the parameter. If the file to be written must be saved in unicode format, you must add a byte header when creating the file.

Cfile file;
Wchar szwbuffer [128];
Wchar * pszunicode = l "Unicode string/N"; // Unicode string
Char * pszansi = "ANSI string/N"; // ANSI string
Word wsignature = 0 xfeff;
File. Open (text ("test.txt"), cfile: modecreate | cfile: modewrite );
File. Write (& wsignature, 2 );
File. Write (pszunicode, lstrlenw (pszunicode) * sizeof (wchar ));
// Explicitly use lstrlenw Function
Multibytetowidechar (cp_acp, 0, pszansi,-1, szwbuffer, 128 );
File. Write (szwbuffer, lstrlenw (szwbuffer) * sizeof (wchar ));
File. Close ();
// The code above is valid in Unicode and non-Unicode environments. Here, Unicode is explicitly used for operations.
In non-Unicode environments, all strings in ANSI format are called by default. In this case, tchar is converted to char type unless wchar is explicitly defined. Therefore, in this environment, if you want to read a Unicode file, you must first move two bytes. Then, you must use multibytetowidechar to convert the read string. After conversion, the string information indicates Unicode data.

In the Unicode environment, all strings in unicode format, that is, wide characters, are called by default. In this case, tchar is converted to wchar, and related API functions also call wide character functions. At this time, the Unicode file is read in the same way as the preceding one, but the data obtained by reading is wchar. to convert it to ANSI format, you need to call widechartomultibyte. If you want to read ANSI data, you do not need to move two bytes to directly read the data and convert the data as needed.

Some languages (such as Korean) must be displayed in the Unicode environment. In this case, the development in a non-Unicode environment cannot achieve the purpose of displaying text even if string function conversion is used, because at this time, the API function is called using ANSI (although the underlying layer uses Unicode processing, the processing result is displayed according to the API called by the programmer ). Therefore, Unicode must be used for development.