Recently, in the MySQL database used by the project team, garbled characters are inserted into the data. We have summarized this issue, starting from the most basic point, to the deep-seated problem
Recently, in the MySQL database used by the project team, garbled characters are inserted into the data. We have summarized this issue, starting from the most basic point, to the deep-seated problem
Recently, in the MySQL database used by the project team, garbled data is inserted. We have summarized this issue. Let's start from the basic point, to the deep-seated causes and solutions of errors.
Basic Concepts
• Character refers to the smallest semantic symbol in human language. For example, 'A' and 'B;
• Given a series of characters, each character is assigned a value, which is used to represent the corresponding character. This value is the character Encoding (Encoding ). For example, if 'A' is given A value of 0 and 'B' is given A value of 1, 0 is the encoding of 'A;
• Given a series of characters and the corresponding encoding, a Set of all these characters and encoding pairs is a Character Set ). For example, if the given character list is {'A', 'B'}, {'A' => 0, 'B' => 1} is A character set;
• Collation refers to the comparison rules between characters in the same character set;
• Only after confirming the character order can an equivalent character set be defined and the relationship between characters in size be defined;
• Each Collation only corresponds to one character set, but one character set can correspond to multiple character sequences, one of which is the Default Collation );
• Names in the collation of MySQL follow naming conventions: names starting with character sets corresponding to the collation; names starting with _ ci (Case Insensitive) and _ cs (case sensitive) or end with _ bin (compare by encoding value. For example, in the collation "utf8_general_ci", "a" and "A" are equivalent;
MySQL Character Set settings
• System variables:
-Character_set_server: Default internal operation Character Set
-Character_set_client: character set used by the client source data
-Character_set_connection: Connection layer Character Set
-Character_set_results: Query Result Character Set
-Character_set_database: Default Character Set of the currently selected Database
-Character_set_system: System metadata (field name, etc.) Character Set
-There are also variables starting with collation _, which are used to describe the collation.
• Use introducer to specify the character set of the text string:
-Format: [_ charset] 'string' [COLLATE collation]
-Example:
SELECT _ latin1 'string ';
SELECT _ utf8 'Hello 'COLLATE utf8_general_ci;
-Text strings modified by introducer are directly converted to internal character sets for processing without extra transcoding during the request.
Character Set conversion process in MySQL
1. MySQL Server converts the request data from character_set_client to character_set_connection when receiving the request;
2. Before performing internal operations, convert the request data from character_set_connection to the internal operation character set. The method is as follows:
-SET the character set value for each data field;
-If the preceding value does not exist, use the default character set Value of the corresponding data table (MySQL extension, non-SQL standard );
-If the preceding value does not exist, use the default character set Value of the corresponding database;
-If the preceding value does not exist, use character_set_server to set the value.
3. Convert the operation result from the internal character set to character_set_results.
Let's go back and analyze the Garbled text:
A our field does not have a character set, so we use the table's Dataset
B. the character set is not specified in our table. The data inventory character set is used by default.
C. The character set is not specified when our database is created. Therefore, use character_set_server to set the value.
D. We didn't specifically modify the specified character set of character_set_server. Therefore, the mysql Default
E mysql Default character set is latin1, so we use the latin1 character set, and our character_set_connection character set is UTF-8, inserting Chinese garbled characters is hard to avoid.
FAQs
• The Connection character set is not set before the FAQ-1 inserts utf8 encoding data into a data table with the default Character Set utf8, and the connection character set is set to utf8 during Query
-The default settings of the MySQL server are used during insertion. character_set_client, character_set_connection, and character_set_results are latin1;
-The data to be inserted will go through the character set conversion process of latin1 => latin1 => utf8. During this process, each inserted Chinese character will be changed from the original three bytes to 6 bytes for storage;
-The query result will go through the utf8 => utf8 Character Set conversion process, and the 6 bytes saved will not be returned, resulting in garbled characters. Refer:
• Before inserting utf8 encoded data into a data table with the default Character Set latin1, set the connection character set to utf8 (this is the error we encountered)
-During insertion, character_set_client, character_set_connection, and character_set_results are set to utf8;
-- The inserted data is converted to the character set utf8 => utf8 => latin1. If the raw data contains \ u0000 ~ Unicode characters outside the \ u00ff range will be converted to "?" because they cannot be expressed in the latin1 character set. (0 × 3F). The content cannot be restored regardless of the character set setting. The conversion process is as follows:
Methods for detecting Character Set Problems
• Show character set;
• Show collation;
• Show variables like 'character % ';
• Show variables like 'collation % ';
• SQL functions HEX, LENGTH, and CHAR_LENGTH
• SQL functions CHARSET and COLLATION
Suggestions for using MySQL character sets
• When creating databases/Tables and performing database operations, try to explicitly specify the character set used, instead of relying on the default settings of MySQL. Otherwise, MySQL upgrades may cause great problems;
• When latin1 is used for databases and character sets, garbled characters can be solved in most cases, but the disadvantage is that SQL operations cannot be performed in characters, generally, setting the database and connected character set to utf8 is a good choice;
• When using mysql CAPI (mysql provides APIs for C language operations), you can use mysql_options to SET the MYSQL_SET_CHARSET_NAME attribute to utf8 immediately after initializing the database handle. In this way, you do not need to explicitly use the set names statement to specify the connection character SET, when mysql_ping is used to reconnect to a persistent connection, the connection character set is reset to utf8;
• For mysql PHP APIs, page-level PHP programs generally run for a short period of time. After connecting to the database, you can explicitly use the set names statement to SET a character SET for connection. However, when using a persistent connection, note that the connection is smooth and the set names statement is used to explicitly reset the character SET after reconnection.
Other considerations
• Default_character_set in my. cnf only affects the connection character set when the mysql command is used to connect to the server, and does not affect applications that use the libmysqlclient library!
• SQL function operations on fields are generally performed using the internal operation character set, which is not affected by the character set setting of the connection.
• The bare strings in SQL statements will be affected by the connected character set or introducer settings. For comparison and other operations, they may produce completely different results. Be careful!
Summary
According to the above analysis and suggestions, we should be clear about how to solve our problems. Yes, it is to specify the character set when creating the database. do not modify the default configuration to achieve the goal. Of course, you can also use the character set of the specified table, but it is easy to miss out, especially when many people participate in the design, it is more likely to leak out.
Although it is not recommended to modify the default Character Set of mysql, I still provide some methods for you to modify the default character set.
MySQL Default Character Set
MySQL can specify character sets to a database, a table, and a column. traditional programs do not use such complex configurations when creating databases and data tables. They use default configurations.
(1) When compiling MySQL, a default character set is specified, which is latin1;
(2) When installing MySQL, you can specify a default character set in the configuration file (my. ini). If this character set is not specified, this value inherits from the one specified during compilation;
(3) When mysqld is started, you can specify a default character set in the command line parameters. If not, this value inherits from the configuration in the configuration file, character_set_server is set to the default character set;
(4) install MySQL select multi-language support, setup will automatically set default_character_set in the configuration file to UTF-8, ensure that by default all columns of all databases and all tables are stored in UTF-8.
View default character sets
(By default, the mysql character set is latin1 (ISO_8859_1). We have provided related commands on how to view it.
Modify default Character Set
(1) The simplest modification method is to modify the character set key value in mysql's my. ini file,
For example, default-character-set = utf8
Character_set_server = utf8
After modification, restart the mysql service.
(2) another way to modify the character set is to use mysql commands.
Mysql> SET character_set_client = utf8;
Mysql> SET character_set_connection = utf8;
Mysql> SET character_set_database = utf8;
Mysql> SET character_set_results = utf8;
Mysql> SET character_set_server = utf8;
Mysql> SET collation_connection = utf8;
Mysql> SET collation_database = utf8;
Mysql> SET collation_server = utf8;
If the default character set for the table is set to utf8 and the query is sent by UTF-8 encoding, garbled characters are still stored in the database. There may be a problem at the connection layer. The solution is to execute the following sentence before sending the query: set names 'utf8'; it is equivalent to the following three commands:
SET character_set_client = utf8;
SET character_set_results = utf8;
SET character_set_connection = utf8;
