) Memory usage in MySQL

MySQL memory management is actually complicated. In summary, there are two types:

1-thread exclusive memory

2. Global shared memory

Thread exclusive memory:

Thread stack information uses memory (thread_stack): it is mainly used to store the identity information of each thread, such as the thread ID and basic information during thread running, we can use the thread_stack parameter to set the size of memory allocated to each thread stack.

Memory Used for sorting (sort_buffer_size): MySQL uses this memory area for sorting (filesort) to complete client sorting requests. When the cache size in the sorting area cannot meet the actual memory size required for sorting, MySQL writes data to disk files for sorting. Because the read/write performance of disks and memory is not an order of magnitude, the effect of sort_buffer_size on the Performance of sorting operations cannot be underestimated. For the implementation principle of sorting operations, see Implementation Analysis of MySQL order.

The Join Operation uses memory (join_buffer_size): applications often have two or multiple tables to join, when MySQL meets certain join requirements (all/index join), in order to reduce the number of reads from the "driven table" involved in the join operation to improve performance, you need to use the JOIN Buffer to complete the join operation. (For more information about the join implementation algorithm, see the basic implementation principle of join in MySQL ). When the JOIN Buffer is too small, MySQL will not store the buffer into a disk file. Instead, it first performs the join operation on the result set in the JOIN Buffer and the table to be joined, and then clears the data in the JOIN Buffer, continue to write the remaining result set into this buffer. This will inevitably cause the driver table to be read multiple times, doubling Io access and reducing efficiency.

Memory Used for sequential data buffer reading (read_buffer_size): this memory is mainly used for full table scanning and full index scanning when data needs to be read in sequence, for example, if no index is available. In this case, MySQL reads data blocks sequentially according to the data storage order. The data read is saved in read_buffer_size first. When the buffer space is full or all data is read, then, the data in the buffer is returned to the upper-layer caller to Improve the efficiency.

The random read data buffer uses the memory (read_rnd_buffer_size): corresponds to the sequential read. when MySQL reads data blocks in an unordered (random read) manner, it uses this buffer to temporarily store the read data. For example, read Table Data Based on index information, and join the table based on the sorted result set. In general, when the reading of data blocks needs to meet certain sequence, MySQL needs to generate random reading and then use the memory buffer set by the read_rnd_buffer_size parameter.

The connection information and the result set before returning the client are temporarily stored in memory (net_buffer_size): This part is used to store the connection information of the client connection thread and return the result set of the client. When MySQL starts to generate a result set that can be returned, it will first save the result set in the buffer set through net_buffer_size before returning it to the client request thread through the network, to improve network transmission efficiency. However, the net_buffer_size parameter only sets the initialization size of the cache area. MySQL will apply for more memory as needed, but the maximum size will not exceed the value of max_allowed_packet.

Bulk_insert_buffer_size: When we use insert... Values (...), (...), (...)... MySQL first places the submitted data in a cache space. When the cache space is full or all data is submitted, mySQL writes data in the cache space to the database at a time and clears the cache. In addition, when we load data infile to load the data in the text file into the database, this buffer is also used.

Temporary table memory usage (tmp_table_size): When we perform some special operations such as order by and group by which temporary tables are required, MySQL may need to use temporary tables. When the temporary table is small (smaller than the size set by the tmp_table_size parameter), MySQL creates the temporary table as a memory temporary table, only when the size set by tmp_table_size cannot be attached to the entire temporary table can Mysql CREATE the table as the MyISAM storage engine and store it on the disk. However, when the size of another system parameter max_heap_table_size is smaller than tmp_table_size, MySQL uses the size set by the max_heap_table_size parameter as the maximum temporary memory table size, while ignoring the value set by tmp_table_size. The tmp_table_size parameter is available only from MySQL 5.1.2, and max_heap_table_size has been used before.

2. Global Sharing

In global sharing, MySQL instance (mysqld process) and underlying storage engine are used to store various global operations and shared temporary storage information, such as query cache for query cache, cache the thread cache of the connection thread, the table cache for caching the table file handle information, and the BINLOG buffer for caching binary logs, cache key buffer of the MyISAM storage engine index key and InnoDB buffer pool that stores InnoDB data and indexes. The following is a simple analysis of MySQL's main shared memory.

Query cache: Query cache is a unique cache region of MySQL. It is used to cache the result set information of a specific query and share it with all clients. After a specific hash calculation is performed on a query statement, it is stored in the query cache corresponding to the result set to improve the speed of the same query statement. After MySQL's query cache is enabled, MySQL receives a SELECT query and obtains the hash value of the query through a fixed hash algorithm, then, go to the query cache to check whether there is a corresponding query cache. If yes, the cache result set is directly returned to the client. If no, perform subsequent operations. After obtaining the corresponding result set, cache the result set to the query cache and return it to the client. When the data of any table changes, all query cache related to the table will be invalid. Therefore, query cache is not very suitable for tables with frequent changes, however, it is very suitable for tables with fewer changes and can greatly improve the query efficiency, such as static resource tables and configuration tables. To use query cache as efficiently as possible, MySQL designs multiple query_cache_type values and two query hints: SQL _cache and SQL _no_cache for query cache. When query_cache_type is set to 0 (or off), query cache is not used. When it is set to 1 (or on, mySQL ignores the query cache only when SQL _no_cache is used in the query. When query_cache_type is set to 2 (or demand, mySQL uses the query cache only when the SQL _cache prompt is used in the query. You can use query_cache_size to set the maximum memory space that can be used.

Connection thread cache: to improve the efficiency of creating connection threads, MySQL keeps some idle connection threads in a cache area for new connection requests, this can greatly improve the efficiency of creating connections, especially for applications that use transient connections. After we set the size of the connection thread that can be cached by the connection thread cache pool through thread_cache_size, we can calculate the hit rate of the connection thread cache through (connections-threads_created)/connections * 100%. Note that the number of connection threads that can be cached is set here, rather than the memory size.

Table cache: The table cache area is used to cache file handle information of table files. In versions earlier than mysql5.1.3, the table cache parameter is set, but from mysql5.1.3 to table_open_cache to set its size. When our client program submits a query to MySQL, MySQL needs to obtain a table file handle information for each table involved in the query. If there is no table cache, therefore, MySQL has to open and close files frequently, which will undoubtedly affect the system performance. Table cache is generated to solve this problem. When MySQL needs to obtain the handle information of a table file after table cache is available, it first goes to table cache to check whether there are idle table file handles. If yes, it is used directly. If no, you can only open the file to obtain the file handle information. After use, MySQL puts the file handle information back in the table cache pool for other threads to use. Note that the number of table file handle information that can be cached is set here, rather than the size of memory space.

Table definition cache (table definition cache): It is a new cache zone introduced from mysql5.1.3 to store table definition information. When many tables are used in MySQL, this cache will undoubtedly improve the access efficiency of table definition information. MySQL provides the table_definition_cache parameter to set the number of tables that can be cached. In versions earlier than mysql5.1.25, the default value is 128. From mysql5.1.25, the default value is adjusted to 256. The maximum value is 524288. Note that the number of table definitions that can be cached is set here, rather than the size of memory space.

Binary log Buffer: the binary log buffer is mainly used to cache binary log information generated by various data changes. To improve system performance, MySQL does not directly write binary logs to log files every time, but first writes information to BINLOG buffer. when certain conditions are met (such as sync_binlog parameter settings) and then write it into the log file again. We can use binlog_cache_size to set the memory size that can be used, and use max_binlog_cache_size to limit its maximum size (MySQL will apply for more memory when a single transaction is too large ). When the required memory is greater than the max_binlog_cache_size parameter, MySQL reports the error "Multi-statement transaction required more than 'max _ binlog_cache_size 'bytes of storage ".

MyISAM index cache (key buffer): The MyISAM index cache caches the index information of the MyISAM table in the memory to improve its access performance. This cache is one of the most important factors affecting the performance of the MyISAM storage engine. You can use key_buffere_size to set the maximum memory space that can be used.

InnoDB log Buffer: the buffer used by transaction logs of the InnoDB Storage engine. Similar to BINLOG buffer, InnoDB writes information to innofb log buffer to improve performance when writing transaction logs, when the corresponding conditions set by the innodb_flush_log_trx_commit parameter are met (or the log buffer is full), the log will be written to a file (or synchronized to the disk. You can use the innodb_log_buffer_size parameter to set the maximum memory space that can be used.

Note: The innodb_flush_log_trx_commit parameter has a critical impact on the InnoDB log write performance. This parameter can be set to 0, 1, and 2. The explanation is as follows:

0: Data in the log buffer is written to the log file at a frequency every second, and the file system is synchronized to the disk at the same time, however, the commit of each transaction does not trigger any refresh from log buffer to log file or from the file system to disk;

1: Data in the log buffer will be written to the log file during each transaction commit, and synchronization from the file system to the disk will also be triggered;

2: transaction commit triggers refresh from log buffer to log file, but does not trigger Disk File System to disk synchronization. In addition, a file system is synchronized to the disk every second.

In addition, the MySQL documentation also mentions that the mechanism for synchronizing these settings once per second may not completely ensure that synchronization will occur every second very accurately, but also depends on the process scheduling problem. In fact, whether InnoDB can really meet the meaning of the value set by this parameter indicates that normal recovery is still restricted by file systems and disks in different operating systems, sometimes, if the Disk Synchronization is not completed, mysqld is also notified that the disk synchronization has been completed.

InnoDB Buffer Pool: The Role of InnoDB buffer pool on the InnoDB Storage engine is similar to that of the key buffer cache on the MyISAM storage engine, the main difference is that the InnoDB buffer pool not only caches index data, but also caches table data, and completely caches the data according to the fast data structure information in the data file, this is very similar to the database buffer cache in Oracle SGA. Therefore, the InnoDB buffer pool has a big impact on the performance of the InnoDB Storage engine. You can use (innodb_buffer_pool_read_requests-innodb_buffer_pool_reads)/innodb_buffer_pool_read_requests * 100% to calculate the InnoDB buffer pool hit rate.

InnoDB dictionary information cache (InnoDB additional memory pool): InnoDB dictionary information cache is mainly used to store dictionary information of the InnoDB Storage engine and some internal shared data structure information. Therefore, the size of the table depends on the number of InnoDB Storage engine tables used in the system. However, if the memory size we set through the innodb_additional_mem_pool_size parameter is not enough, InnoDB will automatically apply for more memory and record warning information in the MySQL error log.