Code Analysis for MHA failover and online failover

Code Analysis for MHA failover and online failover

Some time ago, my colleague Shen Longxing sorted out the MHA failover and online failover code processes and forwarded them here after obtaining their consent. This article is based on MySQL5.5. Therefore, gtid is not involved. MHA Master-slave switchover can be divided into two types: failover and rotate. The former is applicable to the case where the original Master is down, and the latter is used for online failover. The following describes

Failover processing process

1. MHA: MasterFailover: main ()
   
2. -> Do_master_failover
   
3. Phase 1: Configuration Check Phase
   
4. -> Check_settings:
   
5. Check_node_version: View MHA version information
   
6. Connect_all_and_read_server_status: Check whether the MySQL instances of each node can be connected.
   
7. Get_dead_servers/get_alive_servers/get_alive_slaves: double check the status of each node
   
8. Start_ SQL _threads_if: Check whether Slave_ SQL _Running is Yes. If not, start SQL thread
   
9. 
   
10. Phase 2: Dead Master Shutdown Phase: for us, the only function is to stop IO thread
    
11. -> Force_shutdown ($ dead_master ):
    
12. Stop_io_thread: stop all slave IO threads (stop master)
    
13. Force_shutdown_internal (in fact, it is to execute master_ip_failover_script/shutdown_script in the configuration file. If not, it will not be executed ):
    
14. Master_ip_failover_script: If the VIP is set, switch the VIP first.
    
15. Shutdown_script: If the shutdown script is set, run
    
16. 
    
17. Phase 3: Master Recovery Phase
    
18. -> Phase 3.1: Getting Latest Slaves Phase (obtain latest slave)
    
19. Read_slave_status: obtains the binlog file/position of each slave.
    
20. Check_slave_status: Call "show slave status" to obtain the following slave information:
    
21. Slave_IO_State, Master_Host,
    
22. Master_Port, Master_User,
    
23. Slave_IO_Running, Slave_ SQL _Running,
    
24. Master_Log_File, Read_Master_Log_Pos,
    
25. Relay_Master_Log_File, Last_Errno,
    
26. Last_Error, Exec_Master_Log_Pos,
    
27. Relay_Log_File, Relay_Log_Pos,
    
28. Seconds_Behind_Master, Retrieved_Gtid_Set,
    
29. Executed_Gtid_Set, Auto_Position
    
30. Replicate_Do_DB, Replicate_Ignore_DB, Replicate_Do_Table,
    
31. Replicate_Ignore_Table, Replicate_Wild_Do_Table,
    
32. Replicate_Wild_Ignore_Table
    
33. Identify_latest_slaves:
    
34. Compare Master_Log_File/Read_Master_Log_Pos in each slave to find the latest slave
    
35. Identify_oldest_slaves:
    
36. Compare Master_Log_File/Read_Master_Log_Pos in each slave to find the oldest slave
    
37. 
    
38. -> Phase 3.2: Saving Dead Master's Binlog Phase:
    
39. Save_master_binlog:
    
40. If the dead master can be connected through ssh, the following branches are used:
    
41. Save_master_binlog_internal :( copy the save_binary_logs script of the node on the dead master)
    
42. Save_binary_logs -- command = save -- start_file = mysql-bin.000281 -- start_pos = 107 -- binlog_dir =/opt/mysql/data/binlog -- output_file =/opt/mha/log/logs -- handle_raw_binlog = 1 -- disable_log_bin = 0 -- manager_version = 0.55.
    
43. Generate_diff_binary_log:
    
44. Concat_all_binlogs_from:
    
45. Dump_binlog: dumps the binlog file to the target file and uses binmode read.
    
46. Dump_binlog_header_fde: Read position-1 from 0
    
47. Dump_binlog_from_pos: Starting from position, dump binlog file to target file
    
48. File_copy:
    
49. Copy the binlog file generated above to the manager_workdir directory of the manage node.
    
50. If the dead master cannot log on via ssh, txn not synchronized to slave on the master is lost.
    
51. 
    
52. -> PHP 3.3: Determining New Master Phase
    
53. Find_latest_base_slave:
    
54. Find_latest_base_slave_internal:
    
55. Pos_cmp ($ oldest_mlf, $ oldest_mlp, $ latest_mlf, $ latest_mlp)
    
56. Determine whether the binlog location of latest/oldest slave is the same. If the location is the same, the relay log does not need to be synchronized.
    
57. Apply_diff_relay_logs -- command = find -- latest
    
58. Check whether there is a relay log missing from oldest in latest slave. If not, continue; otherwise, failover fails.
    
59. The search method is very simple, that is, reading the latest slave relay log file in reverse order until the file/position is found.
    
60. 
    
61. Select_new_master: selects a new master node.
    
62. If preferred node is specified, one of active preferred nodes will be new master.
    
63. If the latest server behinds too much (I. e. stopping SQL thread for online backups ),
    
64. We shocould not use it as a new master, we shocould fetch relay log there. Even though preferred
    
65. Master is configured, it does not become a master if it's far behind.
    
66. Get_candidate_masters:
    
67. Is the node configured with candidate_master> 0 in the configuration file.
    
68. Get_bad_candidate_masters:
    
69. # The following servers can not be master:
    
70. #-Dead servers
    
71. #-Set no_master in conf files (I. e. DR servers)
    
72. #-Log_bin is disabled
    
73. #-Major version is not the oldest
    
74. #-Too much replication delay (the binlog position difference between slave and master is greater than 100000000)
    
75. Searching from candidate_master slaves which have received the latest relay log events
    
76. If not found:
    
77. Searching from all candidate_master slaves
    
78. If not found:
    
79. Searching from all slaves which have stored ed the latest relay log events
    
80. If not found:
    
81. Searching from all slaves
    
82. 
    
83. -> PHP 3.4: New Master Diff Log Generation Phase
    
84. Recover_relay_logs:
    
85. Determine whether the new master is latest slave. If not, use the apply_diff_relay_logs command to generate a differential log,
    
86. And sent to the new master
    
87. Recover_master_internal:
    
88. Send the binlog generated in daed master 3.2 to the new master
    
89. 
    
90. -> PHP 3.5: Master Log Apply Phase
    
91. Recover_slave:
    
92. Apply_diff:
    
93. 0. wait_until_relay_log_applied. Wait until the new master finishes executing the relaylog.
    
94. 1. Determine Exec_Master_Log_Pos = Read_Master_Log_Pos,
    
95. If not, use save_binary_logs -- command = save to generate a differential log.
    
96. 2. Call the apply_diff_relay_logs command to make the new master recover. where:
    
97. 2.1 recover logs are divided into three parts:
    
98. Exec_diff: difference between Exec_Master_Log_Pos and Read_Master_Log_Pos
    
99. Read_diff: difference between relay log of new master and lastest slave
    
100. Binlog_diff: binlog difference between lastest slave and daed master
     
101. In fact, apply_diff_relay_logs calls mysqlbinlog command for recover.
     
102. // If the vip is set, you need to call master_ip_failover_script to perform the vip failover.
     
103. 
     
104. Phase 4: Slaves Recovery Phase
     
105. -> Phase 4.1: Starting Parallel Slave Diff Log Generation Phase
     
106. Generate the difference log between Slave and New Slave, and copy the log to the working directory of each Slave.
     
107. 
     
108. -> Phase 4.2: Starting Parallel Slave Log Apply Phase
     
109. Recover_slave:
     
110. Restore each slave, same as Phase3.5
     
111. Change_master_and_start_slave:
     
112. Use the change master to command TO direct the Slave TO the New Master and start the replication (start slave)
     
113. 
     
114. Phase 5: New master cleanup phase
     
115. Reset_slave_on_new_master
     
116. Clearing the New Master is actually resetting the slave info, that is, canceling the original Slave information. So far, the entire Master failover process has been completed.

Rotate processing process

1. MHA: MasterRotate: main ()
   -> Do_master_online_switch:
   Phase 1: Configuration Check Phase
   -> Identify_orig_master
   Connect_all_and_read_server_status:
   Connect_check: First connect check to ensure that MySQL services of each server are normal.
   Connect_and_get_status: Obtain the server_id, mysql_version, log_bin, and other information of the MySQL instance.
   This step also plays an important role in obtaining the current master node. Run show slave status,
   If the output is empty, the current node is a master node.
   Validate_current_master: obtains information about the master node and judges whether the configuration is correct.
   Check whether there is a server down. If yes, exit rotate.
   Check master alive or not, and exit rotate if dead
   Check_repl_priv:
   Check whether the user has the replication permission
   Obtain monitor_advisory_lock to ensure that no other monitor processes are running on the master.
   Run: SELECT GET_LOCK ('mha _ Master_High_Availability_Monitor ',?) AS Value
   Obtain failover_advisory_lock to ensure that no other failover processes are running on the slave.
   Run: SELECT GET_LOCK ('mha _ Master_High_Availability_Failover ',?) AS Value
   Check_replication_health:
   Run: show slave status to determine the following STATUS: current_slave_position/has_replication_problem
   Has_replication_problem check details: IO thread/SQL thread/Seconds_Behind_Master (1 s)
   Get_running_update_threads:
   Use show processlist to query whether the current update thread exists. If yes, exit switch.
   -> Identify_new_master
   Set_latest_slaves: The current slave nodes are all latest slave
   Select_new_master: selects a new master node.
   If preferred node is specified, one of active preferred nodes will be new master.
   If the latest server behinds too much (I. e. stopping SQL thread for online backups ),
   We shocould not use it as a new master, we shocould fetch relay log there. Even though preferred
   Master is configured, it does not become a master if it's far behind.
   Get_candidate_masters:
   Is the node configured with candidate_master> 0 in the configuration file.
   Get_bad_candidate_masters:
   # The following servers can not be master:
   #-Dead servers
   #-Set no_master in conf files (I. e. DR servers)
   #-Log_bin is disabled
   #-Major version is not the oldest
   #-Too much replication delay (the binlog position difference between slave and master is greater than 100000000)
   Searching from candidate_master slaves which have received the latest relay log events
   If not found:
   Searching from all candidate_master slaves
   If not found:
   Searching from all slaves which have stored ed the latest relay log events
   If not found:
   Searching from all slaves
   
   Phase 2: Rejecting updates Phase
   Reject_update: lock table to reject write binlog
   If the "master_ip_online_change_script" parameter is set in the MHA configuration file, run this script to disable writes on the current master.
   This script must be set only when vip is used.
   Reconnect: ensure that the current connection with the master is normal
   Lock_all_tables: Execute flush tables with read lock to lock the table
   Check_binlog_stop: shows the master status twice in a row to determine whether the binlog write has stopped.
   
   Read_slave_status:
   Get_alive_slaves:
   Check_slave_status: Call "show slave status" to obtain the following slave information:
   Slave_IO_State, Master_Host,
   Master_Port, Master_User,
   Slave_IO_Running, Slave_ SQL _Running,
   Master_Log_File, Read_Master_Log_Pos,
   Relay_Master_Log_File, Last_Errno,
   Last_Error, Exec_Master_Log_Pos,
   Relay_Log_File, Relay_Log_Pos,
   Seconds_Behind_Master, Retrieved_Gtid_Set,
   Executed_Gtid_Set, Auto_Position
   Replicate_Do_DB, Replicate_Ignore_DB, Replicate_Do_Table,
   Replicate_Ignore_Table, Replicate_Wild_Do_Table,
   Replicate_Wild_Ignore_Table
   Switch_master:
   Switch_master_internal:
   Master_pos_wait: Call the select master_pos_wait function and wait for the master-slave synchronization to complete.
   Get_new_master_binlog_position: Run 'show master status'
   Allow write access on the new master:
   Call master_ip_online_change_script -- command = start... to point the vip to the new master.
   Disable_read_only:
   Run: set global read_only = 0 on the new master.
   Switch_slaves:
   Switch_slaves_internal:
   Change_master_and_start_slave
   Change_master:
   Start_slave:
   Unlock_tables: Execute unlock table on orig master
   Phase 5: New master cleanup phase
   Reset_slave_on_new_master
   Release_failover_advisory_lock