Tachyon Framework's Worker heartbeat and Master High Availability Analysis, tachyonworker

Tachyon Framework's Worker heartbeat and Master High Availability Analysis, tachyonworker
0 Overview

The Master-Slave type in the distributed framework. The Slave node is responsible for the specific execution of the work, and the Master is responsible for task distribution or storage of related metadata. Generally, a Master node corresponds to multiple Slave nodes, when assigning tasks, the Master needs to know which Slave nodes can accept their own commands (the Slave node may be suspended for various reasons ), therefore, you need to maintain a linked list inside it to save all the living Slave nodes. The HMaster of HBase is like this, the NameNode of HDFS is like this, and the Master node of Tachyon is also like this. The server Load balancer node communicates with the Master node through HeartBeat. The Master node regards the Server Load balancer node that receives the HeartBeat report as currently alive. Otherwise, the Server Load balancer node fails. In addition to maintaining the memory activity, the Master node usually returns the commands to be executed to the Slave node through heartbeat. The Slave node receives the commands sent by the Master to complete an interaction.

Master is the core. If it fails, it will have a fatal impact on the entire system. single point of failure (spof) is a problem that should be considered by every distributed framework. Zookeeper that applies and implements the Paxos algorithm is a powerful tool to solve the consistency problem. HDFS, Storm, and HBase all use Zookeeper as the carrier of metadata information HA, and Tachyon is no exception.

1 Worker heartbeat 1.1 overall process

The Worker node of Tachyon reports the memory used in the current Worker and the data block information to the Master through heartbeat, after the Master receives the heartbeat report, it returns the execution commands related to the Worker node. These commands can be Register, Free, or Nothing.

Tachyon. worker. to. master. heartbeat. interval. ms settings.

(2) The timeout value of the Worker heartbeat parameter is set to 10 seconds by default by tachyon. worker. heartbeat. timeout. ms.

(3) There are five main Commands returned by the Master to the Worker: Unknown, Nothing, Register, Free, and Delete. The Nothing command does Nothing. The Register command executes the Worker registration to the Master, and the Master returns the WorkerId stored locally in the Worker; the Free command releases the data stored in the Worker memory; the Delete command deletes both the data in the memory and the data on the disk.

(4) The heartbeat method of WorkerStorage is called.

(5) CheckStatus checks the memory usage managed by the current Worker node.

1.2 normal HeartBeat Processing

 Step 1: Call the cleanConnect method to disable the transport port of thrift and stop the current HeartbeatThread. if the object is not null, set mIsShutdown to true, TException is thrown. For the first clean operation, there is almost nothing to do.

Step 2: Clean, enter the while loop, and prepare to obtain the master Address to establish a connection. The while LOOP condition is tries ++ <MAX_CONNECT_TRY &&! MIsShutdown: retry five times by default.

Step 3: Get the current address of the Master. Use the getMasterAddress method to find all nodes in the leader directory in ZK. Based on the Creation Time of the node, find the latest node as the Active Master node to be connected.

Step 4: Initialize the communication protocol between the Thrift client and the server. TBinaryProtocol is used here.

Step 5: Initialize the Thrift Client MasterService. Client object.

Step 6: To enable the Protocol's Transport, it is the data transmission channel, ready to read and write. If it fails to open, it will throw TTransportException, and then stop the HeartbeatThread used to maintain the connection, in addition, after sleep1 second, the attempt to retry the while operation fails to reach the maximum number of failures, and a TException is thrown.

Step 7: Initializes HeartbeatThread, which continuously performs heartbeat with the server. The timeout value is tachyon. user. master. client. timeout. the attribute value configured by ms. The default value is 10 seconds. This parameter is used to keep the created Thrift connection alive. If the heartbeat times out, the cleanConnect method is called and the Thrift data transmission channel is disabled, terminate this heartbeat thread.

(4) After the connect connection is successful, the worker_heartbeat method of MasterService. client is called for heartbeat processing. The result returned is Command. The worker_heartbeat process is as follows:

Step 1: Call the send_worker_heartbeat method to set the workerId, memory used, and blockId deleted by the worker.

Step 2: Create a worker_heartbeat_result object. worker_heartbeat_result is a static internal class in MasterService. Two types of fields are defined here. The value of SUCCESS_FIELD_DESC is 0 and that of E_FIELD_DESC is 1.

Step 3: It is sent to the Master through the Thrift service and handled by the workerHeartbeat method of MasterInfo. If the Worker node cannot find the Worker information reported by heartbeat, The CommandType. Register command is returned to the Worker node. If memory needs to be released, the Master returns the CommandType. Free command to the Worker node. Otherwise, the CommandType. Nothing command is returned to the Worker node.

1.3 HeartBeat Exception Handling

Two major exceptions may occur during TachyonWorker heartbeat reporting: BlockInfoException and TException exception.

(1) If BlockInfoException occurs, the WorkerStorage. checkStatus () method is called in TachyonWorker. If the heartbeat condition is still true, that is, the mStop parameter is false, the heartbeat report is continued.

(2) If TException occurs, call the WorkerStorage resetMasterClient method in TachyonWorker to reset the MasterClient object and connect to the Thrift server using the connect method. It should be noted that the Worker heartbeat timeout determines that the default timeout time is 10 seconds. If the timeout time is reached, a RuntimeException is thrown, and the heartbeat thread crashes directly. If no heartbeat timeout occurs, then proceed to the checkStatus of WorkerStorage, re-check the heartbeat condition, and enter the next heartbeat.

2 Master HA

The Master node will create the Journal directory during initialization. If the underlying file system is HDFS, it will directly create the corresponding directory on HDFS, and you need to Format it (the formatting here is actually to create an empty file to mark the Format ).

The file system information of Tachyon is stored by Edits logs and Fsimage images (respectively, image. data File and log. data File), Edits Log is the incremental Log of the metadata information of the Tachyon file system, and Fsimage is a snapshot at a certain time point. When the Tachyon Master is started, it first reads the metadata information of the file system from the Fsimage file, that is, information of various data nodes (files, directories, Raw tables, checkpoints, dependencies, etc, then, read the incremental operation records from continuing Edits (possibly multiple). The Edits log content basically corresponds to some operations related to the Tachyon File System Client, including file addition and deletion, rename and add data blocks. However, the Edits log here does not include the actual file content data, but only metadata information. When the file content in the Cache is lost, it is not persistent, and related lineage information is not bound, the contents of the corresponding file will be lost. After this is done, the Tachyon Master will first write the current metadata information into a new Fsimage.

When Zookeeper is used as the HA implementation mechanism of the Master, the Master in the Standby role regularly merges Editlog and creates the Fsimage of Standby. If the Master without Standby is only started, the new Fsimage is generated by merging EditsLog.

Active Master election is completed through the LeaderSelectorClient class. If the current Master is elected as a Leader, the EditsLog rolling is stopped and the init method of MasterInfo is called to initialize relevant parameters, then, start the Web Service UIWebServer (the Standby status Master does not have the WebUI Service), initialize the Master service processing object MasterServiceHandler, and start the Thrift service.

Note that:

(1) The JOURNAL path, the prefix of the formatted file, and the ip addresses and ports of the service and web are all set in the MasterConf class;

(2) The init method of MasterInfo will do the following in sequence:

Step 1: Load the EditsLog file to the memory

Step 2: Create a new image file

Step 3: Create a New EditsLog Log File

Step 4: Create a heartbeat reporter MasterInfoHeartbeatExecutor and start it.

Step 5: Create a file loss restorer RecomputationScheduler and start

(3) The Master node also has a heartbeat, but it only performs periodic system status checks.

Step 1: Get the list of expired workers BlockingQueue, and delete it from the worker storage list on the Master end

Step 2: Try to restore the lost files from the timeout worker list

Step 3: Restart the list of all workers that have timed out. This is important!

(4) The Master creates two nodes in Zookeeper, namely the election node and the face node under it is CreateMode. the temporary node of the EPHEMERAL_SEQUENTIAL type. The leader node, under which the subnode is CreateMode. PERSISTENT type, used to maintain information about the currently active Master node.

(5) because the current Active Master may change, when the Worker selects the Master for communication, it must first retrieve all Master nodes from the leader directory in Zookeeper for traversal, if there is only one Master, it will be returned directly. Otherwise, the Master with the largest cTime can be found as the current Active node, and the Worker node is responsible for communicating with it. While the Worker side will retry five times during heartbeat reporting and throw an exception TException if it still fails. The TachyonWorker's run method catch calls the resetMasterClient Method for resetting, the latest Active Master address will be obtained from Zookeeper every time you connect. If you still cannot connect to the master after a period of time, stop the heartbeat and call the cleanConnect method.

 

-------------------------------------------------------------------------------

If you have read this blog and want to learn more, click[Recommended]

If you want to repost this blog,Please specify the source

If you have any comments or suggestions for this article, please leave a message.

Thank you for reading this article. Please follow up on my blog