The encoding problem in the application is a headache. This is the case all the time. This afternoon, it is a headache caused by database coding errors.
Therefore, we are determined to sum up the problems brought about by coding and strive to give ourselves a clear understanding of the entire coding system.
Starting from the generation of coding problems
We know that computers are invented by Americans, and their English systems have always had only 26 letters, numbers, and special characters. In order to store text information, therefore, the earliest ASCII code is used for character encoding. Later, due to the popularity of computers and the importance of multi-language texts, the characteristics of multi-language became a necessity for computers. countries carried out the National Standard code for each country, and the Chinese version was gb2312 (1980 ), then, in 1995, the Chinese character encoding extension specification (GBK) was promulgated. GBK is compatible with gb2312, but some compatible Chinese characters are added, facilitating conversion with big5 codes. This GBK encoding has gradually become the mainstream encoding for Chinese computers.
Unicode Character Set and UTF-8 Encoding
With the development of computers, a software is often compatible with not only the languages of a country, but also the languages of many countries, especially the three countries in Asia, Japan, and South Korea. The commonly used Chinese characters only contain 7000, therefore, to solve the encoding problem of so many characters, we need to use more scale representation. Unicode is a huge character set that covers languages around the world, in this way, the encoding is more unified and convenient. Unicode does not specify how these characters should be stored in the computer. Although the character set is complete, the efficiency and convenience should be taken into account in the computer, UCS-2 is one of the common solutions, using two bytes to encode a character, which is not compatible with ASCII code, and there is a big problem, the UCS-2 does not represent all Chinese characters, chinese Simplified and Traditional Chinese add up to a total of 60 thousands or 70 thousands, UCS-2 only 65536 encoding, can not save, so only the income of most of the commonly used words. There is also a scheme to represent all Chinese characters UCS-4, but 4 bytes to store Chinese characters, efficiency is relatively low.
UTF is the abbreviation of "UCS Transformation Format" and is an efficient implementation method of the Unicode Character Set.
UTF-8 is a kind of encoding I personally like, it is a variable-length character encoding for Unicode, also known as Wanguo code. It uses 1-6 bytes for encoding, the first 128 encoding content is compatible with ASCII code, so we use a UTF-8 to write pure English text is almost the same as ASCII code. (Note that I am talking about almost ).
If Unicode characters are represented by 2 bytes, encoding into a UTF-8 may take 3 bytes. If Unicode characters are represented by four bytes, it may take six bytes to encode them into a UTF-8. There may be too many Unicode characters encoded with four or six bytes, but such Unicode characters are often uncommon.
The idea of this encoding is similar to that of Harman encoding, which shortens the high-frequency characters and the low-frequency characters, making the overall encoding efficiency better.
I used to mix Unicode and UTF-8 for a while, Unicode is a character set, not strict on the Internet also refers to the commonly used encoding UCS-2 of the Unicode Character Set, that is, Unicode codes encoded in two bytes.
Encoding of some common systems:
VC 6.0 ASCII code
Vs2008-2013 UCS-2 (after Unicode is enabled)
Windows GBK
Linux (andorid) UTF-8
Mac ox UTF-8
Java UCS-2
Python2 ASCII (byte)
Python3 UCS-2
Note: The default encoding is used in the operating system. The system encoding of the three mainstream operating systems can be changed.
MBCS (Multi-Byte Character Set) and code table (codePage)
Next we will introduce the more in-depth concepts in the two character sets, MBCS and codePage.
MBCS (Multi-byte chactacter System) is a general term for all multi-byte encoding solutions. MBCS programming is mainly used for international applications. Because there is usually only one text for a country's market, in order to save resources, these words are often saved in double byte or as few bytes as possible.
Because these double-byte texts are mixed with ANSI, in order to distinguish between them, windows sets the maximum position of these characters to 1 (that is, each byte of these double-byte text> = 127). Therefore, this notation can represent more than 127x127 non-ANSI characters, it can represent common texts in any language. Therefore, Windows has developed separate text encoding for each region version, which is MBCS (Multi-byte code ).
These paging encoding methods are saved into different codepages (internal code table, which indicates the internal code of the machine. Compared with external code, the external input text is encoded, for example, Pinyin, five strokes, and Zheng Code. For example, Chinese is the well-known cp936. Note that this encoding method is exclusive to earlier Windows systems. It is widely used because it was used earlier, cp936 and GB2312-80 are almost the same in coding (see Wikipedia encyclopedia-Chinese character internal code extension specifications), and later expanded GBK, cp936 also carried out the same extension.
The use of this technology dates back to MS-DOS3.3 (released in April 1987) introducing the concept of an inner code table to ibpc users, which is also used by windows.
The internal code table has become an important tool for the system to perform multi-language encoding and conversion. Of course, Unicode code is also included in codePage.
To put it bluntly, MBCS and codePage are the general methods used by computers to solve the problem of multi-byte encoding. The ing of the corresponding encoding can be found in various regional codes, unicode and UTF-8 can also find the corresponding ing in it.
Common codePage
932-Japanese
936-Simplified Chinese (GBK)
949-Korean
950-Traditional Chinese (five major codes)
1200-UCS-2LE Unicode small-end order
1201-UCS-2BE Unicode large-end order
1252-Western European Latin letter ISO-8859-1.
65000-UTF-7 Unicode
65001-UTF-8 Unicode
How does a computer display text?
The computer needs to display the text after storing it, and our LCD screen is composed of pixels, which must be rendered and drawn, send to the video card for raster and display operations.
DOS is the simplest, And the ASCII code can be output in a dot matrix using the motherboard BIOS. Simply, we will not talk much about it. In Windows, how does one draw and display text?
Font Library
We all know that in addition to text encoding, the display font also requires the style of words used for display. This is the font library. Everyone should be familiar with the fonts folder in windows, changing or deleting a font library is just moving files. There are actually different types of font libraries, and the building principles are also different.
The earliest font library is a dot matrix font. This font looks similar to black and white images, that is, it records whether pixels are black or white. We use the built-in font editing program in windows, this font can be processed.
The disadvantage of this font is also obvious. First, it is difficult to zoom in and out. It is easy to distortion when displaying small fonts and Large fonts, And it occupies a large space.
Vector contour font, which is stored Based on the vector principle. It abstracts external boundaries into mathematical vector line segments for convenient scaling and rotation operations. The disadvantage is that the continuity is not good. After amplification, the trip turns into a crease, and the effect is not ideal.
Curve Contour font, which is a method of constructing text through the combination of straight lines and curve segments. It is often used to fit the contour using the bezr curve. The effect is very good, and the font is smooth but the computation is time-consuming.
TrueType technology
TrueType uses Quadratic B-spline Curves and straight lines in ry to describe the shape and contour of the font. Some mathematical algorithms are used to generate large and small fonts. No matter how large or small the font is, the characters are always smooth.
TrueType font, PostScript font, and opentype font are the three main computer Vector Fonts (also known as contour font and stroke font) with the suffix. TTF.
Although the computation is faster, the quality is lower than that of the PostScript font, especially when the text is too small and not clear enough.
Therefore, the font makes a dot matrix font for the small font and the commonly used printed font size to ensure its accuracy. In other cases, the TrueType font is used.
Woff-webopen font format (. woff)
Woff (web development font format) is a specialized font format standard designed for the Web. It is actually an encapsulation of TrueType/OpenType and other font formats, each font file contains the font and metadata (metadata) for the font. The font file is compressed for network transmission.
As a matter of fact, we will find that the font of icons on the network is actually generated based on the most basic font technology.
Font display process
We should have a general understanding of the entire font drawing process.
The first step is to load the text files stored in the hard disk with the corresponding encoding through character encoding and decoding, such as Java file IO, into a string object in the memory, that is, the data that meets the character string storage characteristics of the language. (If you want to, you can also read the data as a binary string and then convert the encoding ), when plotting, the corresponding codePage is called to search for the encoding index, find the corresponding font-shaped index, and then obtain the data in the font Base Based on the font index, based on the method provided by the system to draw a curve to draw a spline for graphic rendering, so that the image on the display can be obtained. More advanced is to reverse the font, sharpen, and perform more detailed operations.
Encoding Problems
The solution to the coding problem is actually very simple, that is, unified encoding. All the areas that need attention are unified encoding formats. In this way, the program will naturally operate according to the encoding.
Web Page code
A large number of user interfaces are used for websites. The encoding of these user interfaces is the core of the entire page code. If you are using PHP, you must output the code on the page, use the header function to encode the page returned by HTTP.
For example:
Header ("Content-Type: text/html; charset = UTF-8 ");
The underlying meaning of this operation is to specify the encoding format for the HTML communication protocol header.
Meta Tags with the same functions
<Meta http-equiv = "Content-Type" content = "text/html; charset = UTF-8">
In fact, it also allows the HTTP protocol to specify the encoding format to prevent garbled characters.
Resource file encoding
Many people think that if the file encoding method is specified, no garbled code will be sent. Is that true? Obviously not, because you should pay attention to how your HTML files are stored? If you use a text editor to save the HTML content as GBK encoding, although it is normal in the text editor, however, because the encoding of your file and the decoding method specified by HTML are not uniform, garbled characters will inevitably occur.
Therefore, you must always note that the storage format of the file determines the encoding format and the declared decoding format. What is even more difficult to understand is that PHP encoding has a lot to pay attention to. For example, if PHP is used as a class code, the encoding of PHP programs is not that important, however, if you define a Chinese string in a program or use PHP as a template engine and a mix of HTML, you should pay attention to the encoding of PHP and its output, which part of the encoding and decoding format does not match will cause garbled characters.
In addition, when using PHP to access files, it is particularly important that the format of the resource files affects the encoding. If you access the files in string mode, but if you store them in binary format, the text encoding is not automatically converted. Therefore, you must manually set the file encoding of such resource files.
Database code
Database encoding is very important. When we use a database, we usually specify the default encoding format when creating a database to prevent garbled characters, but even so, databases often do not work very well, because there is a detail that is hard to be noticed, that is, the encoding of Database SQL statements.
For example:
<Property name = "url" value = "JDBC: mysql: // localhost: 3306/email? Useunicode = true & characterencoding = UTF-8 "/>
The main purpose of this operation is to ensure that the encoding format of the string is unified at the connection layer, regardless of the internal encoding of the database, the returned result is also UTF-8, so that even if the encoding stored in the database is not UTF-8, it will not be affected.
Summary of coding problems in Network Programming
