Hadoop Learning Note Four---Introduction to the Hadoop System communication protocol

This article has agreed:
Dn:datanode
Tt:tasktracker
Nn:namenode
Snn:secondry NameNode
Jt:jobtracker



This article describes the communication protocol between the Hadoop nodes and the client.

Hadoop communication is based on RPC, a detailed introduction to RPC you can refer to "Hadoop RPC mechanism && introduce Avro into the Hadoop RPC mechanism"

Communication between nodes in Hadoop is a more complex network, and if you can understand the communication network between them, it will be helpful to understand the overall framework of Hadoop.


There are 5 types of protocols in HDFs:
Datanodeprotocol (DN && NN)
Interdatanodeprotocol (DN && DN)
Clientdatanodeprotocol (Client && DN)
ClientProtocol (Client && NN)
Namenodeprotocol (SNN && NN)

There are 3 types of protocols in Map/reduce:
Intertrackerprotocol (TT && JT)
Jobsubmissionprotocol (Client && JT)
Taskumbilicalprotocol (Child && TT)

Among them, Datanodeprotocol,clientdatanodeprotocol,intertrackerprotocol,taskumbilicalprotocol, Jobsubmissionprotocol These 5 kinds of protocols communicate frequently.

These 8 protocols exist as interfaces in Hadoop, and 8 protocols inherit the interface Versionedprotocol.
The implementation of the Protocol is mainly concentrated in the class: Jobtracker, NameNode, Tasktracker, DataNode 4 classes.

which
Jobtracker implements two interfaces: Intertrackerprotocol,jobsubmissionprotocol
Tasktracker class implementation: Taskumbilicalprotocol

Namenode class implementation: Datanodeprotocol,clientprotocol,namenodeprotocol
Datanode class implementation: Interdatanodeprotocol,clientdatanodeprotocol

Class Diagram Reference





Each of these protocols is described separately below



1.InterTrackerProtocol (Tasktracker & jobtracker Communication protocol)
Introduction to the Agreement:
TT sends Heartbeat to JT every once in a while to submit status information such as memory, CPU, disk, etc. of the current TT machine, and whether or not it can receive information such as new tasks; TT can obtain the current file system path from JT, need to execute the job's jar file path, etc.

Method Description:
Heartbeat ()//tt periodic (the shortest time default time is 3 seconds) call heartbeat send to JT Heartbeat

Input parameters:

Tasktrackerstatus:tasktracker Machine status information

Free disk information, virtual and actual memory information, map using memory, reduce using memory, can virtual and physical memory, cumulative CPU time, CPU frequency, CPU processor count, CPU utilization

Restarted: Is it the first time you start

Initialcontact: If initialization is required

Acceptnewtasks: Can I receive a new task (Map/reduce)

Responseid: The last ID successfully received from JT

Return Result: (Jobtracker return heartbeatresponse)

The Heartbeatresponse object has several properties:

Responseid: Return answer ID

Heartbeatinterval:heartbeat interval

Action list: What JT needs to do with TT (Launch_task,kill_task,kill_job, Reinit_tracker,commit_task)

Getfilesystemname:tasktracker called once to get the file system path
REPORTTASKTRACKERERROR:TT reporting problems to JT
Gettaskcompletionevents: Get Jobid Task completion information
Getsystemdir: The path to the file required to execute the job

2. Jobsubmissionprotocol (communication between Jobclient & Jobtracker)
Introduction to the Agreement:
Mainly used to submit, execute job and get current job status;
Jobclient can submit the job (SUBMITJOB) through this protocol, get the current job parameters, status, counter,task status, file system path, jar file path, all job status, and job queue information,
In addition, Jobclient can also submit killjob and set job priority tasks to jobtracker;

Main methods:
Getnewjobid: Assigning Jobid
SubmitJob: Submit job execution, return job status
Getclusterstatus: Gets the status of the current cluster, if detailed is true also reports tracker names information, return cluster status
Killjob:kill Job
Setjobpriority: Setting the job priority
Killtask:kill Task attempt
Getjobprofile: Getting a handle to a job that is already running
Getjobstatus: Gets the status of the running job
Getjobcounters: counter to get job
Getmaptaskreports: Individual map task reports for job acquisition
Getreducetaskreports: Each reduce task report that gets a job
Getcleanuptaskreports: Get cleanuptask Results Report
Getsetuptaskreports: Get the Setup Task report
Getfilesystemname: Returns the name of the file system
Jobstocomplete: Get job status with no completion and no failure (running job)
GetAllJobs: Get all Job status
Gettaskcompletionevents: Get a specific Jobid completed event
Getsystemdir: Get the path to the job file storage
Getqueues: Getting the Jobtracker queue collection
Getqueueinfo: Get dispatch information for a job
Getjobsfromqueue: Gets all the submitted jobs in a queue


3. Taskumbilicalprotocol (Child & Tasktracker)
Introduction to the Agreement:
Communication protocol between child & Tasktracker:
Child through this protocol, to see whether TT exists, get tasks, report the progress of tasks, status, error messages, commit files to HDFs, and get map results to reduce;tasktracker receive tasks and monitor the progress of the task

Main methods:
Gettask:child runtime, call this method to get task tasks
Statusupdate: Reporting the progress of the child process
Reportdiagnosticinfo:
Reportnextrecordrange: report the range of records that the task will process next
Ping:child periodically calls this method to see if the parent (Tasktracker) exists
Commitpending: report that this task task is completed, but the file has not yet been committed (not yet commit)
Cancommit: Whether the polling task can commit a file (commit)
The Getmapcompletionevents:reduce task calls this method to get the map output file and return to the Map status update (map completion transaction)

4. Datanodeprotocol (NN & DN)
Introduction to the Agreement:
DN Register information to NN, send current dn,block and other information to NN (send block report, send block error report)
NN return DN action required (delete block or copy)

Main methods:
Register: Registering DN to nn
SENDHEARTBEAT:DN report NN, presence and current status; NN returns the DN Datanodecommand array, telling the DN what to do (Mark block invalidation, copy, etc.)
Blockreport:blockreport () tells Namenode all the local block lists, Namenode returns an obsolete list of blocks that need to be deleted, and this function notifies the namenode of the block on its own machine, and when Hadoop starts, This method namenode the relationship table of block and DN, and other cases are checked periodically (the default time is one months).
Blockreceived:datanode tells Namenode that the recently received block (if there are additional blocks that exceed the number of repetition factors is deleted) For example: The client writes a new block, or another DN copies a block to this DN, this method is called
Errorreport: report namenode data node error
Processupgradecommand: Send Upgrade command to Namenode, return update command
Reportbadblocks: report the wrong block
Nextgenerationstamp: Returns a new timestamp for a piece
Commitblocksynchronization:commit block synchronization in lease recovery

5. Interdatanodeprotocol (DN & DN)
Introduction to the Agreement:
Get Blockmeta, update block information, etc.

Main methods:
Getblockmetadatainfo: Get a Blockmeta information
Updateblock: Update block information (mainly timestamp and length)



6.NamenodeProtocol (NN & SNN)
Introduction to the Agreement:
Get a list of DN blocks, roll back the edit log, roll back the Fsimage

Main methods:
Getblocks: Gets the blocklist of a datanode and is a fixed size, returning a block and an array of positions
Geteditlogsize: Gets the size of the current edit log
Rolleditlog: Close the old edit log and open a new edit log (return Checkpointsignature: A separate transaction ID)
Rollfsimage: The main work to delete the old fsimage, copy the new image to Fsimage; Delete old edits, rename to edits

7.ClientDatanodeProtocol (Client && DN)
Introduction to the Agreement:
Recoverblock: Restore block, return to new blockid and generation stamp


8.ClientProtocol (Client & NN)
Introduction to the Agreement:
Client-->nn: interface of user for file operation, file system operation, System management and problem handling


Main methods:
Getblocklocations: Get a list of the DN of a block within a range of files (sort is sorted by the proximity of the client, and then the client goes directly to the nearest DN based on the DN location to get the actual data),
Create: Creating a new file
Append: Append content to the end of a file (Dfs.support.append works when True)
Setreplication: Sets the recurrence factor for the current file
SetPermission: Set permissions for existing files/directories
SetOwner: Sets the owner of a path
Abandonblock: Discard a block
Addblock: Write block in file
Complete: End Write Data if the file is gracefully closed true; otherwise false
Reportbadblocks:client report a damaged block to Namenode
Rename: Renaming a file or directory
Delete: Deletes a file or directory
Mkdirs: Create a directory
Getlisting: Returns the file directory status
Renewlease: Determine if the client is dead
GetStats: Get the status of filesystem
Getdatanodereport: Get the current Datanode report
Getpreferredblocksize: Block size to get the file
Setsafemode: Enter Safe mode
Savenamespace: Save namespace image
Refreshnodes:namenode re-reading the Hosts file
Finalizeupgrade: End Last update
Metasave: Dumps the current system Namenode data structure to a file, if present append
GetFileInfo: Obtaining information about a file
Setquota: Set quotas for file directories
Getcontentsummary: Content Summary of the catalog
Fsync: Sync files all metadata to file
Settimes: Set file time

-----------------loaded from http://weilaiyxj.iteye.com/blog/913166

Hadoop Learning Note Four---Introduction to the Hadoop System communication protocol