MySQL batch SQL insert performance optimization details, mysql Performance Optimization

MySQL batch SQL insert performance optimization details, mysql Performance Optimization

For some systems with a large data volume, the database faces problems in addition to the low query efficiency, but also the long data warehouse receiving time. Especially for the report system, the time spent on data import may be several hours or dozens of hours every day. Therefore, it is meaningful to optimize the database insertion performance.

Some performance tests on MySQL innodb have found some ways to improve the insert efficiency for your reference.

1. Insert multiple data records into one SQL statement.

Common insert statements are as follows:

INSERT INTO `insert_table` (`datetime`, `uid`, `content`, `type`)   VALUES ('0', 'userid_0', 'content_0', 0); INSERT INTO `insert_table` (`datetime`, `uid`, `content`, `type`)   VALUES ('1', 'userid_1', 'content_1', 1); 

Modify:

INSERT INTO `insert_table` (`datetime`, `uid`, `content`, `type`)   VALUES ('0', 'userid_0', 'content_0', 0), ('1', 'userid_1', 'content_1', 1); 

The modified insert operation can improve the insert efficiency of the program. The second high SQL Execution efficiency is mainly caused by the reduction of the merged log volume (MySQL binlog and innodb Transaction reduce the log volume), reducing the data volume and frequency of log disk flushing, and thus improving the efficiency. By combining SQL statements, you can also reduce the number of SQL statement resolutions and reduce the I/O of network transmission.

Here we provide some test and comparison data, namely, importing a single piece of data and converting it into an SQL statement, respectively, testing 1 million, 1 thousand, and 10 thousand data records.

2. Insert a transaction.

Modify the insert statement:

START TRANSACTION; INSERT INTO `insert_table` (`datetime`, `uid`, `content`, `type`)   VALUES ('0', 'userid_0', 'content_0', 0); INSERT INTO `insert_table` (`datetime`, `uid`, `content`, `type`)   VALUES ('1', 'userid_1', 'content_1', 1); ... COMMIT; 

The use of transactions can improve data insertion efficiency, because during an INSERT operation, MySQL will establish a transaction internally to perform real insertion processing in the transaction. You can use transactions to reduce the consumption of creating transactions. All inserts are committed only after execution.

The test comparison is also provided here, where no transaction is used and the number of transactions used is 10 thousand, and respectively.

3. Data is inserted in sequence.

Data insertion means that the inserted records are sorted on the primary key. For example, datetime is the primary key of the record:

INSERT INTO `insert_table` (`datetime`, `uid`, `content`, `type`)   VALUES ('1', 'userid_1', 'content_1', 1); INSERT INTO `insert_table` (`datetime`, `uid`, `content`, `type`)   VALUES ('0', 'userid_0', 'content_0', 0); INSERT INTO `insert_table` (`datetime`, `uid`, `content`, `type`)   VALUES ('2', 'userid_2', 'content_2',2); 

Modify:

INSERT INTO `insert_table` (`datetime`, `uid`, `content`, `type`)   VALUES ('0', 'userid_0', 'content_0', 0); INSERT INTO `insert_table` (`datetime`, `uid`, `content`, `type`)   VALUES ('1', 'userid_1', 'content_1', 1); INSERT INTO `insert_table` (`datetime`, `uid`, `content`, `type`)   VALUES ('2', 'userid_2', 'content_2',2); 

Because index data needs to be maintained during database insertion, unordered records will increase the cost of index maintenance. We can refer to the B + tree index used by innodb. If each inserted record is at the end of the index, the index positioning efficiency is high and the index adjustment is small; if the inserted records are in the middle of the index, the B + tree Splitting and merging operations will consume a lot of computing resources, and the index positioning efficiency of the inserted records will decrease, disk operations are frequently performed when the data volume is large.

The following describes the performance comparison between random data and ordered data, which are recorded as 10 thousand, 0.1 million, 1 million, and respectively.

From the test results, the performance of the optimization method has been improved, but the improvement is not very obvious.

Comprehensive Performance test:

Here we provide a test to optimize the INSERT efficiency by using the above three methods at the same time.

From the test results, we can see that when the data merging + transaction method has a small amount of data, the performance improvement is obvious. When the data volume is large (more than 10 million), the performance will drop sharply, this is because the data volume exceeds the innodb_buffer capacity at this time. Each index location involves a large number of disk read/write operations, and the performance decreases rapidly. However, the use of combined data + transactions + ordered data still performs well when the data volume reaches more than 10 million levels. When the data volume is large, it is easier to locate ordered data indexes, you do not need to perform read/write operations on disks frequently, so you can maintain high performance.

Note:

1. the SQL statement has a length limit. when data is merged in the same SQL statement, the length limit cannot be exceeded. The max_allowed_packet configuration can be modified. The default value is 1 MB and the value is 8 Mb.

2. The transaction size needs to be controlled. Too large a transaction may affect the execution efficiency. MySQL has the innodb_log_buffer_size configuration item. If this value is exceeded, innodb data will be flushed to the disk, and the efficiency will decrease. Therefore, it is better to commit a transaction before the data reaches this value.

The above is all the content of this article. I hope it will be helpful for your learning and support for helping customers.