MySQL source code analysis (7): MyISAM Data File Processing-reprint

Document directory

• Basic knowledge
• Record processing interface
• Fixed type

I haven't written an analysis article for a long time. One is busy, and the other is because the rest of the content is hard and it takes time to understand it. The remaining interesting content is:

• Execution and optimization of select statements. You are concerned about the query performance of the database, especially the query optimization part.
• Mysql replication. The master/slave architecture of Mysql is the best choice for most high-performance website architectures using mysql. replication is the basis of this architecture.
• Implementation of specific database engines. This part is also a part of interest to many people who are concerned about mysql performance. However, this work is complicated, especially the popular innodb, which is extremely heavy and difficult. The transaction section is particularly complicated.

In addition, I found that my article was excerpted from some places. Thank you for reading it. But I also hope to indicate the source in the abstract. At least give a link to the original article, I am lucky enough.

Today, I mainly write Myisam data file processing.

Myisam is the first Mysql database engine to be implemented, and is also the engine with the best performance in people's minds (although it is not the most powerful, there is no way, the reality often requires a balance between performance and functionality ). Here, I chose to analyze it mainly because its implementation is relatively simple and clear, and recently I am interested in the format of data files, especially the processing of variable-length data. Note that this article does not introduce the index file format of myisam.
Basic knowledge

For each table that uses Myisam as a data engine, the following files are saved in the <% data_dir %>/<database> directory:

• . Frm file. This file is cross-engine and describes the metadata of the table. The most important is the table definition and the database engine of the table.
• . MYD file. This is the key file to be viewed. It contains the database record information, that is, each row in the database.
• . MYI file. Index file to accelerate search.

Each record in MYD can be fixed, dynamic, or packed. Fixed indicates that the record size is fixed and there is no such thing as VARCHAR or blob. Dynamic is the opposite, with a variable-length data type. The packed type isMyisampackThe processed record. See http://dev.mysql.com/doc/refman/5.1/en/myisam-table-formats.html. Note that the record type is for table settings, rather than for each column.
Record processing interface

The record type is set at the table level. Therefore, when a table is opened, myisam checks the metadata options to see what type of record the table is, then, set the corresponding number of processing functions. The specific processing is in the mi_setup_functions of storage/myisam/mi_open.c. Let's look at one of the fragments:

746 void mi_setup_functions (register myisam_share * share) 747 {.... 759 else if (share-> options & packages) 760 {761 share-> read_record = _ mi_read_dynamic_record; 762 share-> read_rnd = _ blank; 763 share-> delete_record = _ mi_delete_dynamic_record; 764 share-> compare_record = _ mi_cmp_dynamic_record; 765 share-> compare_unique = _ mi_cmp_dynamic_unique; 766 share-> calc_checksum = mi_checksum; 767768/* Add bits used to pack data to pack_reclength for faster allocation */769 share-> base. pack_reclength + = Share-> base. pack_bits; 770 if (share-> base. blobs) 771 {772 share-> update_record = _ mi_update_blob_record; 773 share-> write_record = _ mi_write_blob_record; 774} 775 else776 {777 share-> write_record = _ records; 778 share-> update_record = _ mi_update_dynamic_record; 779} 780 }...

This is the processing function setting for the pack type. Set a bunch of function interfaces in the share structure. By the way, this method is commonly used in C Programming to Implement "polymorphism": declare the function interface and dynamically set the interface implementation, the idea is consistent with the dynamic binding of C ++. This Code sets the record processing function for a dynamic table. It is interesting that HA_OPTION_PACK_RECORD is used to specify the dynamic type. We can see what these function names are doing. Let's take a look at the specific processing of fixed and dynamic types.

As the name suggests, all fields in a Fixed table are fixed and cannot contain TEXT, VARCHAR, and other things. The advantage of such strict restrictions is faster and more direct data record operations. Think about it and you know that every data is fixed and convenient for file operations.
Look at the data function _ mi_write_static_record. In mi_statrec.c, all the implementations of fixed record operations are defined in this file. 21 int _ mi_write_static_record (mi_info * info, const uchar * record) 22 {... 24 if (Info-> S-> state. dellink! = Ha_offset_error & 25! Info-> append_insert_at_end) 26 {check whether record exists in dellink. Dellink is a linked list composed of all deleted data. When a record is deleted, the file size occupied by it is not immediately released, but put into dellink for the next use. 27 my_off_t filepos = Info-> S-> state. dellink; read information about the data space pointed to by dellink. 33. Update dellink to remove the used data space. Write the record to the space where the deleted data is found. 40} 41 else42 {43 check whether the data file is too large. 49 If the write buffer is used, the write buffer is used. Write new data to the end of the file. Update metadata. ... 86}

Because all data is of the same size, it is easy to process. Especially when a data is deleted, the space occupied by the data is put into a recycle linked list. If the recycle linked list is not empty when new data is to be written next time, you can directly find a new data written from it, without allocating a new storage space. Other processes of the fixed type are also very simple. It should be pointed out that no matter what type of data is used, when the data is deleted, the space occupied by the data is not immediately released, so the operation is too costly, it is intolerable to move all the data behind the data forward. The general practice is to wear these spaces with linked lists for future use, so data files are generally not automatically reduced... even InnoDB. Dynamic type

The Dynamic type is relative to the fixed type. This type can tolerate the existence of variable-length data types. It is followed by more complex operations on data files. The deleted data blocks in the Dynamic type are not immediately released or linked. The next time you want to write new data, you should first find it from the linked list. Unlike the fixed type, the size of the new data and the empty space in the linked list may be different. If the new data is big, several free spaces will be found to distribute the data in multiple data blocks. If the new data is small, the free data blocks will be divided into two parts, write new data, and put the data in the free linked list for later users. Let's take a look at the write_dynamic_record function in mi_dynrec.c. 320 static int write_dynamic_record (MI_INFO * info, const uchar * record, 321 ulong reclength) 322 {check whether there is sufficient space to store new data. If the space is full, an error is returned. 351352 do353 {// find a place where data can be written. Note that this is in a loop. That is to say, the entire data may not be written to the space found each time. Only part of the data can be written, and the rest of the data should be written to another place. 354 if (_ mi_find_writepos (info, reclength, & filepos, & length) 355 goto err; // write data that can be stored in the found space. 356 if (_ mi_write_part_record (info, filepos, length, 357 (info-> append_insert_at_end? 358 HA_OFFSET_ERROR: info-> s-> state. dellink), 359 (uchar **) & record, & reclength, & flag) 360 goto err; 361} while (reclength );...} the loop describes everything. It is very likely that a piece of data will be divided into several parts and written to different places, but they constitute the entire data. Look at _ mi_find_writepos. 371 static int _ mi_find_writepos (MI_INFO * info, 372 ulong reclength,/* record length */373 my_off_t * filepos,/* Return file pos */374 ulong * length) /* length of block at filepos */375 {376 MI_BLOCK_INFO block_info ;... // check whether there is any available space in dellink. 380 if (info-> s-> state. dellink! = HA_OFFSET_ERROR & 381! Info-> append_insert_at_end) 382 {383/* Deleted blocks exists; Get last used block */There is free space, then find the header in the linked list, use the space to write new data. Return the description of the space to the caller. .... 398} 399 else400 {401/* No deleted blocks; Allocate a new block */No deleted space, then Allocate space at the end of the data file and return it to the caller. 421 }...

} If the deleted space exists, the space described in the linked list header is directly returned. This algorithm is very simple, but I think this simple algorithm may cause some problems, such as storage fragmentation, and the size of a large space is getting smaller and smaller, it takes several spaces to write data later. These problems also exist in the memory management of the operating system. Therefore, a large number of memory management algorithms are generated. You can also borrow them here. The specific write is completed in _ mi_write_part_record. This function is relatively long and I can simply name it as follows. Int _ mi_write_part_record (mi_info * info, my_off_t filepos,/* points at empty block */ulong length,/* length of block */my_off_t next_filepos, /* Next empty block */uchar ** record,/* pointer to record PTR */ulong * reclength,/* length of * record */int * flag) /** flag = 0 if header */{if the given space is greater than the data length, calculate the remaining space after the data is filled. If the space is good, prepare some metadata. If the space is too small, locate the location of the next write space (either the next dellink or the end of the file) and prepare the metadata. For the first part of the data, write more information. If the space is too large and there is space available, check whether the space can be connected to the next free space to form a large space. If yes, merge the space. Prepare related metadata, such as space location and size. Start to write data. If write buffering is enabled, write data to the buffer; otherwise, write data to the space found. Updates dellink information .}

The logic is clear, mainly to deal with the complexity brought about by too much space or too small. Well, most of the processing here is very clear, and it is still very straight. The rest is to put the space occupied by a data block in dellink When deleting the data. Note that if the data block can be merged with other data blocks in dellink, the merge operation is also called in the delete data operation, and the merged data block may continue to be merged with other data blocks. If you are interested, check delete_dynamic_record and I will not write it.