Go deep into Mysql Character Set settings. Before reading this article, let's take a look at the articles in the snow. For more information, see
Go deep into Mysql Character Set settings. Before reading this article, let's take a look at the articles in the snow. For more information, see
Basic Concepts
• Character refers to the smallest semantic symbol in human language. For example, 'A' and 'B;
• Given a series of characters, each character is assigned a value, which is used to represent the corresponding character. This value is the character Encoding (Encoding ). For example, if 'A' is given A value of 0 and 'B' is given A value of 1, 0 is the encoding of 'A;
• Given a series of characters and the corresponding encoding, a Set of all these characters and encoding pairs is a Character Set ). For example, if the given character list is {'A', 'B'}, {'A' => 0, 'B' => 1} is A character set;
• Collation refers to the comparison rules between characters in the same character set;
• Only after confirming the character order can an equivalent character set be defined and the relationship between characters in size be defined;
• Each Collation only corresponds to one character set, but one character set can correspond to multiple character sequences, one of which is the Default Collation );
• Names in the collation of MySQL follow naming conventions: names starting with character sets corresponding to the collation; names starting with _ ci (Case Insensitive) and _ cs (case sensitive) or end with _ bin (compare by encoding value. For example, in the collation ''utf8 _ general_ci '', ''a' and ''a' are equivalent;
MySQL Character Set settings
• System variables:
-Character_set_server: Default internal operation Character Set
-Character_set_client: character set used by the client source data
-Character_set_connection: Connection layer Character Set
-Character_set_results: Query Result Character Set
-Character_set_database: Default Character Set of the currently selected Database
-Character_set_system: System metadata (field name, etc.) Character Set
-There are also variables starting with collation _, which are used to describe the collation.
• Use introducer to specify the character set of the text string:
-Format: [_ charset] 'string' [COLLATE collation]
-Example:
• SELECT _ latin1 'string ';
• SELECT _ utf8 'Hello 'COLLATE utf8_general_ci;
-Text strings modified by introducer are directly converted to internal character sets for processing without extra transcoding during the request.
Character Set conversion process in MySQL
1. MySQL Server converts the request data from character_set_client to character_set_connection when receiving the request;
2. Before performing internal operations, convert the request data from character_set_connection to the internal operation character set. The method is as follows:
• SET the character set value for each data field;
• If the preceding value does not exist, use the default character set Value of the corresponding data table (MySQL extension, non-SQL standard );
• If the preceding value does not exist, use the default character set Value of the corresponding database;
• If the preceding value does not exist, use character_set_server to set the value.
3. Convert the operation result from the internal character set to character_set_results.
FAQs
• No connection character set is set before UTF-8 encoding is inserted to a data table whose default character set is utf8. During query, the connection character set is set to utf8.
-The default settings of the MySQL server are used during insertion. character_set_client, character_set_connection, and character_set_results are latin1;
-The data to be inserted will go through the character set conversion process of latin1 => latin1 => utf8. During this process, each inserted Chinese character will be changed from the original three bytes to 6 bytes for storage;
-The query result will go through the utf8 => utf8 Character Set conversion process, and the 6 bytes saved will not be returned, resulting in garbled characters ......
• Set the connection character set to utf8 before inserting utf8-encoded data into a data table with the default Character Set latin1
-During insertion, character_set_client, character_set_connection, and character_set_results are set to utf8;
-The inserted data is converted to the character set utf8 => utf8 => latin1. If the raw data contains/u0000 ~ Unicode characters outside the/u00ff range will be converted to "?" because they cannot be expressed in the latin1 character set. (0x3F) symbol. In future queries, the content cannot be restored regardless of the character set setting.
Methods for detecting Character Set Problems
• Show character set;
• Show collation;
• Show variables like 'character % ';
• Show variables like 'collation % ';
• SQL functions HEX, LENGTH, and CHAR_LENGTH
• SQL functions CHARSET and COLLATION
Suggestions for using MySQL character sets
• When creating databases/Tables and performing database operations, try to explicitly specify the character set used, instead of relying on the default settings of MySQL. Otherwise, MySQL upgrades may cause great problems;
• Although latin1 can be used in databases and character sets to solve garbled characters in most cases, the disadvantage is that SQL operations cannot be performed in characters, generally, setting the database and connected character set to utf8 is a good choice;
• When using mysql c api, you can use mysql_options to SET the MYSQL_SET_CHARSET_NAME attribute to utf8 immediately after initializing the database handle, so that you do not need to explicitly use the set names statement to specify the character SET to be connected, when mysql_ping is used to reconnect to a persistent connection, the connection character set is reset to utf8;
• For mysql PHP APIs, page-level PHP programs generally run for a short period of time. After connecting to the database, you can explicitly use the set names statement to SET a character SET for connection. However, when using a persistent connection, note that the connection is smooth and the set names statement is used to explicitly reset the character SET after reconnection.
Other considerations
• Default_character_set in my. cnf only affects the connection character set when the mysql command is used to connect to the server, and does not affect applications that use the libmysqlclient library!
• SQL function operations on fields are generally performed using the internal operation character set, which is not affected by the character set setting of the connection.
• The bare strings in SQL statements will be affected by the connected character set or introducer settings. For comparison and other operations, they may produce completely different results. Be careful! /P>
There is a character set converter between the client and the server. Character_set_client => gbk: the converter knows that the client sends the gbk encoding character_set_connection => gbk: converts the data sent from the client to the gbk character_set_results => gbk:
Note:The preceding three character sets can be set using set names gbk. Example: create table test (name varchar (64) not null) charset utf8; # utf8 indicates the character encoding on the server. First, insert a data inert into test values ('test') to the data table test. Then, the data "test" is stored in the database in the "utf8" format:
First, the data is sent to the mysql server through the Mysql client. When the character_set_connection value is gbk, the data sent from the client is converted to gbk format, when the character set converter transmits the data to the server, it finds that the server saves the data by utf8. Therefore, the data is automatically converted from gbk to utf8 within the server.
When Will garbled characters appear?
Client data format and declaredCharacter_set_client does not match
Use the header ('content-type: text/html; charset = utf8'); to convert the client data to utf8 format. When the data passes through the "Character Set converter, because character_set_client = gbk and character_set_connection is equal to gbk, the data transmitted from the client (in fact, in utf8 format) will not be converted.
However, when the character set converter sends data to the server, it finds that the server requires utf8 format, so the current data will be processed as gbk format, and then converted to utf8 (however, this step is actually wrong ...). 2. When the result does not match the client page
Set the format of the returned result to utf8, but the format accepted by the client is gbk, so garbled characters will appear.
Show character set syntax can be used to display all available character sets latin character sets
Note: The Maxlen column displays the maximum number of bytes used to store a single character. Utf8 Character Set
Gbk character set
When Will data be lost?Compared with the above three images, we can know that in each character set, the maximum number of bytes used to store a character is different, utf8 is the maximum, and latin is the smallest. Therefore, when a character set converter is used, improper processing may cause data loss and cannot be recovered. For example:
SetWhen the character_set_connection value is changed to lantin
The gbk data sent from the client is converted to the lantin1 format, because the gbk format occupies a large number of characters, resulting in data loss.
Summary:
Character_set_client and character_set_results must be consistent in general, because one represents the data format sent by the client, and the other represents the data format accepted by the client to avoid data loss, the character_set_connection character encoding must be greater than the character_set_client character encoding.
