Hadoop metrics parameter description

People who use hadoop have some knowledge about the detailed counters in hadoop, but many may not find any information when they want to fully understand all metrics. In addition, there are few introductions when searching in the code. List all items.

DFS. datanode. blockchecksumop_avg_time block verification average time DFS. datanode. blockchecksumop_num_ops block check count DFS. datanode. blockreports_avg_time average time of the block report DFS. datanode. blockreports_num_ops block report count DFS. datanode. block_verification_failures block verification failure count DFS. datanode. blocks_read: Total number of times block read from hard disk DFS. datanode. blocks_removed: Number of deleted blocks DFS. datanode. total number of blocks_replicated block copies DFS. datanode. blocks_verified total number of block verification times DFS. datanode. blocks_written: Total number of times block writing to the hard disk DFS. datanode. bytes_read read Total Bytes include the number of bytes of the CRC verification file DFS. datanode. bytes_written total number of bytes written to the table (the number of bytes written to each packet) DFS. datanode. copyblockop_avg_time: Average copy time (unit: MS) DFS. datanode. copyblockop_num_ops: number of times of copying a block DFS. datanode. heartbeats_avg_time reports the average DFS time to namenode. datanode. heartbeats_num_ops reports the total number of times to namenode DFS. datanode. readblockop_avg_time average read block time (unit: MS) DFS. datanode. readblockop_num_ops: Average number of read blocks and DFS. datanode. consistent blocks_read: First read the input stream from the hard disk to add DFS. datanode. blocks_read count, and then increase the Count DFS. datanode. reads_from_local_client reads data from the Local Machine (DFS. datanode. reads_from_remote_client reads data from remote blocks DFS. datanode. replaceblockop_avg_time replaces the average block time (SLB Policy) DFS. datanode. replaceblockop_num_ops Replace (SLB Policy) DFS. datanode. volumefailures notfound and block-owned volume failure are related to DFS. datanode. writeblockop_avg_time average write block time DFS. datanode. the total number of writeblockop_num_ops write blocks is generally the same as that of DFS. datanode. blocks_written is consistent. Add DFS from the hard disk first. datanode. blocks_read count, and then increase the Count DFS. datanode. writes_from_local_client number of local writes DFS. datanode. writes_from_remote_client number of remote writes JVM. metrics. gccount GC total times JVM. metrics. gctimemillis GC total time consumed (MS) JVM. metrics. logerror JVM error count JVM. metrics. the number of times that logfatal JVM appears fatal JVM. metrics. number of loginfo JVM info occurrences JVM. metrics. the number of times logwarn JVM warn appears JVM. metrics. maxmemorym JVM tries to use the maximum memory (M) and returns long if there is no limit. max_valuejvm.metrics.memheapcommittedm JVM submit heap memory size JVM. metrics. memheapusedm JVM uses heap memory size JVM. metrics. memnonheapcommittedm JVM non-heap memory submitted size JVM. metrics. memnonheapusedm JVM non-heap memory used size JVM. metrics. threadsblocked is blocking the number of threads waiting for the monitor lock JVM. metrics. threadsnew: Number of threads not started JVM. metrics. threadsrunnable the number of threads in the execution state JVM. metrics. threadsterminated Number of exited threads JVM. metrics. threadstimedwaiting: waiting for another thread to execute depends on the number of threads for the specified wait time operation JVM. metrics. threadswaiting the number of threads waiting for another thread to execute a specific operation indefinitely

Rpc. Metrics. numopenconnections number of open connections RPC connections
Rpc. Metrics. receivedbytes number of bytes received by RPC
Rpc. Metrics. rpcprocessingtime_avg_time average time for RPC operations in last interval RPC average operation time in recent interactions
Rpc. Metrics. rpcprocessingtime_num_ops RPC connection count in the most recent interaction
Rpc. Metrics. rpcqueuetime_avg_time average RPC wait time during interaction
Number of RPC Operations completed in rpc. Metrics. rpcqueuetime_num_ops RPC queue
Rpc. Metrics. sentbytes number of bytes sent data bytes sent by RPC
Rpc. Metrics. callqueuelen length of the RPC queue RPC Queue Length
Rpc. Metrics. rpcauthenticationfailures number of failures in RPC Authentication
Rpc. Metrics. rpcauthenticationsuccesses number of successful authentications verified successfully
Rpc. Metrics. rpcauthorizationfailures number of failures in authorization
Rpc. Metrics. rpcauthorizationsuccesses number of successful Successful authorizations

Mapred. shuffleinput. shuffle_failed_fetches
Mapred. shuffleinput. shuffle_fetchers_busy_percent: Percentage of threads busy in obtaining map output in parallel
Mapred. shuffleinput. shuffle_input_bytes read data bytes during shuffle
Mapred. shuffleinput. shuffle_success_fetches
Mapred. shuffleoutput. shuffle_failed_outputs
Mapred. shuffleoutput. shuffle_handler_busy_percent: Percentage of busy server threads in map output sent to reduce (configured in tasktracker. http. threads.
Mapred. shuffleoutput. shuffle_output_bytes output data bytes during shuffle
Mapred. shuffleoutput. shuffle_success_outputs is successfully directed to reduce.
Mapred. tasktracker. maptaskslots: set the number of map Slots
Mapred. tasktracker. maps_running Number of running maps
Mapred. tasktracker. cetcetaskslots: set the number of reduce Slots
Mapred. tasktracker. reduces_running: Number of running reduce tasks
Mapred. tasktracker. tasks_completed
Mapred. tasktracker. tasks_failed_ping the number of failed tasks caused by tasktracker interaction with tasks
Mapred. tasktracker. tasks_failed_timeout Number of kill tasks that time out because tasks are not configured in mapred. task. Timeout (10 minutes by default ).
Rpc. detailed-metrics.canCommit_avg_time RPC ask whether to submit the task Average Time
Rpc. detailed-metrics.canCommit_num_ops RPC ask whether to submit the task count
Rpc. detailed-metrics.commitPending_avg_time RPC Report Task commit completed, but the average time that the submission is still in pending state
Rpc. detailed-metrics.commitPending_num_ops RPC reports the number of times that the job is submitted completed, but it is still in pending state
Rpc. detailed-metrics.done_avg_time RPC reports the average time of successful completion of the task
Rpc. detailed-metrics.done_num_ops RPC reports the number of successful tasks
Rpc. detailed-metrics.fatalError_avg_time RPC reports the average time for the task to experience fatalError
Rpc. detailed-metrics.fatalError_num_ops RPC reports the number of times the task experienced a fatalError
Rpc. Average time the detailed-metrics.getBlockInfo_avg_time gets the block from the specified datanode
Rpc. Number of times the detailed-metrics.getBlockInfo_num_ops gets a block from a specified datanode
Rpc. detailed-metrics.getMapCompletionEvents_avg_time reduce gets the average time of completed map output address events
Rpc. Number of times the detailed-metrics.getMapCompletionEvents_num_ops reduce gets completed map output address events
Average time for rpc. detailed-metrics.getProtocolVersion_avg_time to get RPC Protocol Version Information
Rpc. Number of times the detailed-metrics.getProtocolVersion_num_ops obtains RPC Protocol Version Information
Rpc. detailed-metrics.getTask_avg_time get the average time of jvmtask after the child process starts
Rpc. detailed-metrics.getTask_num_ops the number of jvmtask requests when the child process is started
Rpc. The average time for the detailed-metrics.ping_avg_time sub-process to periodically check whether the parent process is still alive
Rpc. Number of times the detailed-metrics.ping_num_ops sub-process periodically checks whether the parent process is still alive
Rpc. The average time that the detailed-metrics.recoverBlock_avg_time generates for the specified block start recovery tag
Rpc. The number of times the detailed-metrics.recoverBlock_num_ops starts to restore tag generation for the specified Block
Rpc. The average time for the detailed-metrics.reportDiagnosticInfo_avg_time to report task error messages to the parent process, which is saved in jobtracker as little as possible
Rpc. Number of times the detailed-metrics.reportDiagnosticInfo_num_ops reports task error messages to the parent process
Rpc. Average time for the detailed-metrics.startBlockRecovery_avg_time to start block recovery
Rpc. Number of times the detailed-metrics.startBlockRecovery_num_ops starts to recover the block
Rpc. The average time that the detailed-metrics.statusUpdate_avg_time reports the progress of the child process to the parent process
Rpc. The number of times the detailed-metrics.statusUpdate_num_ops reports the progress of the child process to the parent process
Rpc. The detailed-metrics.updateBlock_avg_time updates the block to the new tag and the average operating time of the length
Rpc. Number of times the detailed-metrics.updateBlock_num_ops updates a block to a new tag and length