In-depth understanding of MySQL 5.7 GTID series (II): GTID-related internal data structure, mysqlgtid

In-depth understanding of MySQL 5.7 GTID series (II): GTID-related internal data structure, mysqlgtid

Author:Gao Peng (Eight monsters in Chongqing)

Original article address:

Https://www.jianshu.com/p/5649644fdc13

I have a deep understanding of 10 articles in the MySQL 5.7 GTID series. This is the second article,

Click to view the first article: in-depth understanding of MySQL 5.7 GTID series (1)

This series of articles will be updated from time to time ~

1. Basic GTID format

• Single GTID:

e859a28b-b66d-11e7-8371-000c291f347d:1

The first part is SERVER_UUID, and the latter part is the unique identifier for executing the transaction, usually auto-increment. The data structure GTID is used internally, which will be described later.

• Interval GTID:

e859a28b-b66d-11e7-8371-000c291f347d:1-5

The first part is SERVER_UUID, and the latter part is the unique collection of signs for executing the transaction. The INTERVAL node corresponding to a SIDNO in GTID_SET is used internally, which will be described later.

Ii. Generation of SERVER_UUID

Now that SERVER_UUID is mentioned, we will start to discuss the generation of SERVER_UUID.

SERVER_UUID is actually a 32-byte + 1-byte (/0) string. MySQL will call INIT_SERVER_AUTO_OPTIONS ()Read AUTO. CNFFile. Call GENERATE_SERVER_UUID ()Call to generate a SERVER_ID.

In fact, in this function, we can see that SERVER_UUID is at least related to the following part:

• MySQL Start Time

• Thread LWP-related

• A random memory address

See the code snippet:

Const time_t save_server_start_time = server_start_time; / / for MySQL
The startup time
Server_start_time += ((ulonglong)current_pid << 48) + current_pid; // add Lwp number operation
THD - > status_var. Bytes_sent = (THD ulonglong); // this is a memory pointer

Lex_start (THD);
Func_uuid = new (THD - > mem_root) Item_func_uuid ();
Func_uuid - > fixed = 1;
Func_uuid - > val_str (& uuid); // there is a specific operation in this function

After obtaining this informationItem_func_uuid: val_strIf you are interested in the operation, you can take a closer look and finally generate a SERVER_UUID and copy it to the actual SERVER_UUID as follows:

strncpy(server_uuid, uuid.c_ptr(), UUID_LENGTH);

Call stack frame:

#0  init_server_auto_options () at /root/mysql5.7.14/percona-server-5.7.14-7/sql/mysqld.cc:3810
#1  0x0000000000ec625e in mysqld_main (argc=97, argv=0x2e9af08) at /root/mysql5.7.14/percona-server-5.7.14-7/sql/mysqld.cc:4962
#2  0x0000000000ebd604 in main (argc=10, argv=0x7fffffffe458) at /root/mysql5.7.14/percona-server-5.7.14-7/sql/main.cc:25

Iii. SERVER_UUID internal representation BINARY_LOG: UUID

BINARY_LOG: UUIDYesSERVER_UUIDThe internal representation is actually a 16-byte memory space, as shown below:

 /** The number of bytes in the data of a Uuid. */
  static const size_t BYTE_LENGTH= 16;
  /** The data for this Uuid. */
  unsigned char bytes[BYTE_LENGTH];

SERVER_UUIDAndBINARY_LOG: UUIDCan be converted to each other, in SID_MAPBINARY_LOG: UUIDRepresentsSERVER_UUIDIt is actually its SID.

Iv. Class Structure GTID

This structure is an internal representation of a single GTID. Its core elements include:

 /// SIDNO of this Gtid.
  rpl_sidno sidno;
  /// GNO of this Gtid.
  rpl_gno gno;

Gno is the unique identifier of the transaction, and SIDNO is actuallySERVER_UUIDInternal representationBINARY_LOG: UUID (SID)The value of a query table obtained by using the HASH algorithm. Reference FunctionsSID_MAP: ADD_SIDThis function is based onBINARY_LOG: UUID (SID)Returns a SIDNO.

V. Class Structure SID_MAP

Since the HASH algorithm is mentioned, an internal structure is required to store the entire hash query table. in MySQL, SID_MAP is used as such a structure, it contains a variable array and a HASH query table, which has been provided in the comment. There is only one SID_MAP globally. Memory will be allocated when the GTID module is initialized. The core elements of SID_MAP are as follows:

/// Read-write lock that protects updates to the number of SIDNOs.
mutable Checkable_rwlock *sid_lock;
* *
Array that maps SIDNO to SID; the element at index N points to a
Node with SIDNO N-1.
* /
Preallocated_array < node *, 8, true >
//Array search can save node pointer here
* *
Hash that maps SID to SIDNO.  The keys in this array are of type
Rpl_sid.
* /
Hash ﹐ Sid ﹐ to ﹐ sidno; / / because sid is a data structure whose core stores 16 bytes for the key value of bytes according to the server ﹐ UUID
//The irregular memory space generated by the transformation needs to be quickly located by using hash lookup table
* *
Array that maps numbers in the interval [0, get_max_sidno()-1] to
SIDNOs, in order of increasing SID.
@see Sid_map::get_sorted_sidno.
* /
Preallocated ﹤ array < RPL ﹤ sidno, 8, true > ﹤ sorted; / / an additional array about sidno. The specific function is unknown 

Here we will take a look at the type of the pointer element NODE in the variable array:

 struct Node
  {
    rpl_sidno sidno; //sid hash no
    rpl_sid sid; //sid
  };

In fact, it is a corresponding SIDNO and SID.

6. Class Structure GTID_SET

This structure is a collection of certain types of GTID, suchEXECUTE _GTIDSet,PURGE_ GTIDSet. We know thatEXECUTE _GTIDThe GTID that contains multiple databases, that is, multiple SIDNO, may exist simultaneously, and the GTID range of a database may exist as follows:

| gtid_executed                    | 
3558703b-de63-11e7-91c3-5254008768e3:1-6:20-30,
da267088-9c22-11e7-ab56-5254008768e3:1-34 |

Here3558703b-de63-11e7-91c3-5254008768e3The GNO of has multiple intervals:

• 1-6

• 20-30

In this case, the internal representation should be the method of adding the range linked list to the array. Of course, this is the internal implementation of MySQL. Let's look at the core elements:

* *
Array where the N'th element contains the head pointer to the
intervals of SIDNO N+1.
* /
Preallocated_array < interval *, 8, true > m_intervals; / / each sidno contains an interval single Necklace table, which is connected by the next pointer. This completes the problem of splitting gtid, for example
/// Linked list of free intervals.
Interval * free_intervals; / / the free interval is connected to this linked list
/// Linked list of chunks.
Interval_chunk * chunks; / / a global interval list. All the interval spaces are connected on it

7. join class structure INTERVAL of GTID_SET

It indicates the GTID range as mentioned earlier:

• Da267088-9c22-11e7-ab56-5254008768e3: 1-34

Its internal class is the INTERVAL class structure. Let's take a look at the core elements to understand:

 /// The first GNO of this interval.
    rpl_gno start;
    /// The first GNO after this interval.
    rpl_gno end;
    /// Pointer to next interval in list.
    Interval *next;

It is very simple to add a NEXT pointer to start GNO, marking a range.

8. Class Structure GTID_STATE

This structure is initialized with SID_MAP when the database is started, and is also a global variable.
We are familiar with the following parameters:

• GTID_EXECUTED

• GTID_OWNED

• GTID_PURGED

All of them come from times. Of course, in addition to the common ones above, we also include some other core elements. Let's take a look at them:

/// The Sid_map used by this Gtid_state.
Mutable Sid map * Sid map; / / use Sid map
* *
The set of GTIDs that existed in some previously purged binary log.
This is always a subset of executed_gtids.
* /
Gtid ﹣ set lost ﹣ gtids; / / corresponds to the gtid ﹣ purged parameter, which is generally maintained by MySQL automatically unless the gtid ﹣ purged parameter is set manually
/ *
The set of GTIDs that has been executed and
stored into gtid_executed table.
* /
Gtid_set executed_gtids; / / corresponds to the gtid_executed parameter, which is generally actively maintained by mysql
/ *
The set of GTIDs that exists only in gtid_executed table, not in
binlog files.
* /
Gtid_set gtids_only_in_table;
//Normally, for the main database, the collection is always empty, because the main database can't have a gtid that is only in the mysql.gtid table and no longer in the binlog, but for the slave database, you must turn on log, slave, updates and binlog to achieve this effect,
//Otherwise, the bitlog that does not contain the gtid of relay can only be included in the mysql.gtid table. In this case, gtid set gtids only in table always exists. It will be explained later.
/* The previous GTIDs in the last binlog. */
Gtid > set previous > gtids > logged; / / contains all gtids in binlog that have been executed in the previous binlog
/// The set of GTIDs that are owned by some thread.
Owned gtids owned gtids; / / all gtid sets owned by all threads currently
/// The SIDNO for this server.
RPL? Sidno server? Sidno; / / is the sidno from the SID hash corresponding to server? UUID

9. Class Structure OWNED_GTIDS

This structure contains all GTID sets that are being held by the current thread. If GTID is assigned to the transaction, the GTID and thread number are addedOWNED_GTIDSThenTHD-> OWNED_GTIDPoint to this GTID.

Reference FunctionsGTID_STATE: ACQUIRE_OWNERSHIPIn the final stage of COMMITOWNED_GTIDSRemove reference functionsOWNED_GTIDS: REMOVE_GTIDAnd add itGTID_STATE: EXECUTED_GTIDS.

This process will be described later. Its core elements include:

/// Growable array of hashes.  Prealloced_array<HASH*, 8, true> sidno_to_hash;

In this way, each SIDNO has a HASH structure and its HASH content is:

struct Node
  {
    /// GNO of the group.
    rpl_gno gno;
    /// Owner of the group.
    my_thread_id owner;
  };

Such a struct contains GNO and thread ID.

10. Class Structure GTID_TABLE_PERSISTOR

This structure mainly includes some operationsMySQL. GTID_EXECUTEDTable function interface
It mainly includes:

• Insert the gtid into table.int save (THD * thd, const Gtid * gtid );

• Insert the gtid set into table.int save (const Gtid_set * gtid_set );

• Delete all rows from the table.int reset (THD * thd );

• Fetch gtids from gtid_executed table and store them

  Gtid_executed set.int fetch_gtids (Gtid_set * gtid_set );

• Compress the gtid_executed table completely by employing one or more transactions.int compress (THD * thd );

• Write a gtid interval into the gtid_executed table.
  Int write_row (TABLE * table, const char * sid, rpl_gno gno_start, rpl_gno gno_end );

• Update a gtid interval in the gtid_executed table.
  Int update_row (TABLE * table, const char * sid, rpl_gno gno_start, rpl_gno new_gno_end );

• Delete all rows in the gtid_executed table.int delete_all (TABLE * table );
  These methods are also used to determine when to read/write data.MySQL. GTID. EXECUTED.

11. Internal Structure Diagram

To be able to explain the relationships between these class structures in graphs, I modifyMySQL. GTID_EXECUTEDTable andAUTO. CNFThis interval GTID case is constructed. At the same time, after the code is added to the source codeGet_executed_gtids/get_lost_gtids/get_gtids_only_in_table/get_previus_gtids_loggedOutput to log. However, it is difficult to see such a GTID with intervals online.
Assume that various gtids are as follows after the database is started at a certain time point ():

• 2017-12-12T04: 10: 42.768153Z 0 [Note] gtid_state-> get_executed_gtids:
  Gtid have: 3558703b-de63-11e7-91c3-5254008768e3: 1-35,
  Da267088-9c22-11e7-ab56-5254008768e3: 1-34

• 2017-12-12T04: 10: 42.768191Z 0 [Note] gtid_state-> get_lost_gtids:
  Gtid have: 3558703b-de63-11e7-91c3-5254008768e3: 1-6:20-30,
  Da267088-9c22-11e7-ab56-5254008768e3: 1-34

• 2017-12-12T04: 10: 42.768226Z 0 [Note] gtid_state-> get_gtids_only_in_table:
  Gtid have: 3558703b-de63-11e7-91c3-5254008768e3: 1-6:20-30,
  Da267088-9c22-11e7-ab56-5254008768e3: 1-34

• 2017-12-12T04: 10: 42.768260Z 0 [Note] gtid_state-> get_previus_gtids_logged: Gtid have: Success: 7--35

After the transaction is started, we immediately execute a transaction, which is inORDERED_COMMITThe FLUSH phaseGTID_STATE: ACQUIRE_OWNERSHIPGet a GTID, then it isOWNED_GTIDSSo the figure at this time is as follows:

12. Summary

After learning this section, you can at least learn:

• 1. What is SERVER_UUID, how it is generated, and what rules it follows

• 2. How is GTID internally expressed?

• 3. Relationship between SERVER_UUID and GTID internal representation

• 4. GTID_EXECUTED/GTID_OWNED/GTID_PURGED indicates the specific memory structure to which it corresponds. Of course, these will be mentioned many times in the following articles, which will also deepen your understanding of it.

If you have the ability to read source code, you can continue to learn more based on this framework.

If you have any questions about this article, scan the QR code to add the original author.

Zhishutang

Ye Jinrong and Wu Bingxi jointly build

Leading IT elite training

Industry senior experts work together to customize

MySQL practice/MySQL optimization/Python/SQL Optimization

Several excellent courses

Follow the Technology Development Trend and regularly optimize training teaching plans

Integrates a large number of production cases to meet Enterprises' first-line needs

Community companion learning, one registration, three courses available

Required Courses for DBA and development engineers

Thousands of students have turned around, doubling their salary and improving their positions

The change happened quietly. What are you waiting?

Scan QR code to downloadPreview Video of zhishutang high-quality courses

(MySQL practice/optimization, Python development, and SQL optimization courses)

Password: hg3h