Code Analysis for MHA failover and online failover

Source: Internet
Author: User

Code Analysis for MHA failover and online failover


Some time ago, my colleague Shen Longxing sorted out the MHA failover and online failover code processes and forwarded them here after obtaining their consent. This article is based on MySQL5.5. Therefore, gtid is not involved. MHA Master-slave switchover can be divided into two types: failover and rotate. The former is applicable to the case where the original Master is down, and the latter is used for online failover. The following describes



Failover processing process


  1. MHA: MasterFailover: main ()
  2. -> Do_master_failover
  3. Phase 1: Configuration Check Phase
  4. -> Check_settings:
  5. Check_node_version: View MHA version information
  6. Connect_all_and_read_server_status: Check whether the MySQL instances of each node can be connected.
  7. Get_dead_servers/get_alive_servers/get_alive_slaves: double check the status of each node
  8. Start_ SQL _threads_if: Check whether Slave_ SQL _Running is Yes. If not, start SQL thread

  9. Phase 2: Dead Master Shutdown Phase: for us, the only function is to stop IO thread
  10. -> Force_shutdown ($ dead_master ):
  11. Stop_io_thread: stop all slave IO threads (stop master)
  12. Force_shutdown_internal (in fact, it is to execute master_ip_failover_script/shutdown_script in the configuration file. If not, it will not be executed ):
  13. Master_ip_failover_script: If the VIP is set, switch the VIP first.
  14. Shutdown_script: If the shutdown script is set, run

  15. Phase 3: Master Recovery Phase
  16. -> Phase 3.1: Getting Latest Slaves Phase (obtain latest slave)
  17. Read_slave_status: obtains the binlog file/position of each slave.
  18. Check_slave_status: Call "show slave status" to obtain the following slave information:
  19. Slave_IO_State, Master_Host,
  20. Master_Port, Master_User,
  21. Slave_IO_Running, Slave_ SQL _Running,
  22. Master_Log_File, Read_Master_Log_Pos,
  23. Relay_Master_Log_File, Last_Errno,
  24. Last_Error, Exec_Master_Log_Pos,
  25. Relay_Log_File, Relay_Log_Pos,
  26. Seconds_Behind_Master, Retrieved_Gtid_Set,
  27. Executed_Gtid_Set, Auto_Position
  28. Replicate_Do_DB, Replicate_Ignore_DB, Replicate_Do_Table,
  29. Replicate_Ignore_Table, Replicate_Wild_Do_Table,
  30. Replicate_Wild_Ignore_Table
  31. Identify_latest_slaves:
  32. Compare Master_Log_File/Read_Master_Log_Pos in each slave to find the latest slave
  33. Identify_oldest_slaves:
  34. Compare Master_Log_File/Read_Master_Log_Pos in each slave to find the oldest slave

  35. -> Phase 3.2: Saving Dead Master's Binlog Phase:
  36. Save_master_binlog:
  37. If the dead master can be connected through ssh, the following branches are used:
  38. Save_master_binlog_internal :( copy the save_binary_logs script of the node on the dead master)
  39. Save_binary_logs -- command = save -- start_file = mysql-bin.000281 -- start_pos = 107 -- binlog_dir =/opt/mysql/data/binlog -- output_file =/opt/mha/log/logs -- handle_raw_binlog = 1 -- disable_log_bin = 0 -- manager_version = 0.55.
  40. Generate_diff_binary_log:
  41. Concat_all_binlogs_from:
  42. Dump_binlog: dumps the binlog file to the target file and uses binmode read.
  43. Dump_binlog_header_fde: Read position-1 from 0
  44. Dump_binlog_from_pos: Starting from position, dump binlog file to target file
  45. File_copy:
  46. Copy the binlog file generated above to the manager_workdir directory of the manage node.
  47. If the dead master cannot log on via ssh, txn not synchronized to slave on the master is lost.

  48. -> PHP 3.3: Determining New Master Phase
  49. Find_latest_base_slave:
  50. Find_latest_base_slave_internal:
  51. Pos_cmp ($ oldest_mlf, $ oldest_mlp, $ latest_mlf, $ latest_mlp)
  52. Determine whether the binlog location of latest/oldest slave is the same. If the location is the same, the relay log does not need to be synchronized.
  53. Apply_diff_relay_logs -- command = find -- latest
  54. Check whether there is a relay log missing from oldest in latest slave. If not, continue; otherwise, failover fails.
  55. The search method is very simple, that is, reading the latest slave relay log file in reverse order until the file/position is found.

  56. Select_new_master: selects a new master node.
  57. If preferred node is specified, one of active preferred nodes will be new master.
  58. If the latest server behinds too much (I. e. stopping SQL thread for online backups ),
  59. We shocould not use it as a new master, we shocould fetch relay log there. Even though preferred
  60. Master is configured, it does not become a master if it's far behind.
  61. Get_candidate_masters:
  62. Is the node configured with candidate_master> 0 in the configuration file.
  63. Get_bad_candidate_masters:
  64. # The following servers can not be master:
  65. #-Dead servers
  66. #-Set no_master in conf files (I. e. DR servers)
  67. #-Log_bin is disabled
  68. #-Major version is not the oldest
  69. #-Too much replication delay (the binlog position difference between slave and master is greater than 100000000)
  70. Searching from candidate_master slaves which have received the latest relay log events
  71. If not found:
  72. Searching from all candidate_master slaves
  73. If not found:
  74. Searching from all slaves which have stored ed the latest relay log events
  75. If not found:
  76. Searching from all slaves

  77. -> PHP 3.4: New Master Diff Log Generation Phase
  78. Recover_relay_logs:
  79. Determine whether the new master is latest slave. If not, use the apply_diff_relay_logs command to generate a differential log,
  80. And sent to the new master
  81. Recover_master_internal:
  82. Send the binlog generated in daed master 3.2 to the new master

  83. -> PHP 3.5: Master Log Apply Phase
  84. Recover_slave:
  85. Apply_diff:
  86. 0. wait_until_relay_log_applied. Wait until the new master finishes executing the relaylog.
  87. 1. Determine Exec_Master_Log_Pos = Read_Master_Log_Pos,
  88. If not, use save_binary_logs -- command = save to generate a differential log.
  89. 2. Call the apply_diff_relay_logs command to make the new master recover. where:
  90. 2.1 recover logs are divided into three parts:
  91. Exec_diff: difference between Exec_Master_Log_Pos and Read_Master_Log_Pos
  92. Read_diff: difference between relay log of new master and lastest slave
  93. Binlog_diff: binlog difference between lastest slave and daed master
  94. In fact, apply_diff_relay_logs calls mysqlbinlog command for recover.
  95. // If the vip is set, you need to call master_ip_failover_script to perform the vip failover.

  96. Phase 4: Slaves Recovery Phase
  97. -> Phase 4.1: Starting Parallel Slave Diff Log Generation Phase
  98. Generate the difference log between Slave and New Slave, and copy the log to the working directory of each Slave.

  99. -> Phase 4.2: Starting Parallel Slave Log Apply Phase
  100. Recover_slave:
  101. Restore each slave, same as Phase3.5
  102. Change_master_and_start_slave:
  103. Use the change master to command TO direct the Slave TO the New Master and start the replication (start slave)

  104. Phase 5: New master cleanup phase
  105. Reset_slave_on_new_master
  106. Clearing the New Master is actually resetting the slave info, that is, canceling the original Slave information. So far, the entire Master failover process has been completed.


Rotate processing process
  1. MHA: MasterRotate: main ()
    -> Do_master_online_switch:
    Phase 1: Configuration Check Phase
    -> Identify_orig_master
    Connect_all_and_read_server_status:
    Connect_check: First connect check to ensure that MySQL services of each server are normal.
    Connect_and_get_status: Obtain the server_id, mysql_version, log_bin, and other information of the MySQL instance.
    This step also plays an important role in obtaining the current master node. Run show slave status,
    If the output is empty, the current node is a master node.
    Validate_current_master: obtains information about the master node and judges whether the configuration is correct.
    Check whether there is a server down. If yes, exit rotate.
    Check master alive or not, and exit rotate if dead
    Check_repl_priv:
    Check whether the user has the replication permission
    Obtain monitor_advisory_lock to ensure that no other monitor processes are running on the master.
    Run: SELECT GET_LOCK ('mha _ Master_High_Availability_Monitor ',?) AS Value
    Obtain failover_advisory_lock to ensure that no other failover processes are running on the slave.
    Run: SELECT GET_LOCK ('mha _ Master_High_Availability_Failover ',?) AS Value
    Check_replication_health:
    Run: show slave status to determine the following STATUS: current_slave_position/has_replication_problem
    Has_replication_problem check details: IO thread/SQL thread/Seconds_Behind_Master (1 s)
    Get_running_update_threads:
    Use show processlist to query whether the current update thread exists. If yes, exit switch.
    -> Identify_new_master
    Set_latest_slaves: The current slave nodes are all latest slave
    Select_new_master: selects a new master node.
    If preferred node is specified, one of active preferred nodes will be new master.
    If the latest server behinds too much (I. e. stopping SQL thread for online backups ),
    We shocould not use it as a new master, we shocould fetch relay log there. Even though preferred
    Master is configured, it does not become a master if it's far behind.
    Get_candidate_masters:
    Is the node configured with candidate_master> 0 in the configuration file.
    Get_bad_candidate_masters:
    # The following servers can not be master:
    #-Dead servers
    #-Set no_master in conf files (I. e. DR servers)
    #-Log_bin is disabled
    #-Major version is not the oldest
    #-Too much replication delay (the binlog position difference between slave and master is greater than 100000000)
    Searching from candidate_master slaves which have received the latest relay log events
    If not found:
    Searching from all candidate_master slaves
    If not found:
    Searching from all slaves which have stored ed the latest relay log events
    If not found:
    Searching from all slaves

    Phase 2: Rejecting updates Phase
    Reject_update: lock table to reject write binlog
    If the "master_ip_online_change_script" parameter is set in the MHA configuration file, run this script to disable writes on the current master.
    This script must be set only when vip is used.
    Reconnect: ensure that the current connection with the master is normal
    Lock_all_tables: Execute flush tables with read lock to lock the table
    Check_binlog_stop: shows the master status twice in a row to determine whether the binlog write has stopped.

    Read_slave_status:
    Get_alive_slaves:
    Check_slave_status: Call "show slave status" to obtain the following slave information:
    Slave_IO_State, Master_Host,
    Master_Port, Master_User,
    Slave_IO_Running, Slave_ SQL _Running,
    Master_Log_File, Read_Master_Log_Pos,
    Relay_Master_Log_File, Last_Errno,
    Last_Error, Exec_Master_Log_Pos,
    Relay_Log_File, Relay_Log_Pos,
    Seconds_Behind_Master, Retrieved_Gtid_Set,
    Executed_Gtid_Set, Auto_Position
    Replicate_Do_DB, Replicate_Ignore_DB, Replicate_Do_Table,
    Replicate_Ignore_Table, Replicate_Wild_Do_Table,
    Replicate_Wild_Ignore_Table
    Switch_master:
    Switch_master_internal:
    Master_pos_wait: Call the select master_pos_wait function and wait for the master-slave synchronization to complete.
    Get_new_master_binlog_position: Run 'show master status'
    Allow write access on the new master:
    Call master_ip_online_change_script -- command = start... to point the vip to the new master.
    Disable_read_only:
    Run: set global read_only = 0 on the new master.
    Switch_slaves:
    Switch_slaves_internal:
    Change_master_and_start_slave
    Change_master:
    Start_slave:
    Unlock_tables: Execute unlock table on orig master
    Phase 5: New master cleanup phase
    Reset_slave_on_new_master
    Release_failover_advisory_lock

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.