Startupprogress Start-up tracking analysis for HDFs

Objective

Presumably the start-stop operation of the HDFs cluster is definitely not a strange thing for the users of HDFs. In general, we restart the Cluster service for these 2 reasons: 1). The cluster new configuration item requires a restart of the Cluster service to take effect. 2). The cluster-related jar package program was updated You need to restart the service to run the latest jar package. So we restarted the cluster, the general we see whether it started successfully, will certainly pay attention to its output log. For example, in HDFs, we will pay more attention to Namenode's startup log. Of course, this is an effective way But do we have other ways to be more convenient and efficient? For example, I don't want to go to the Namenode node to look at the log. The final answer is yes. What is the process and principle in this paper? This is the topic that this article will focus on: the startupprogress of HDFs.

Startupprogress of HDFs

Here you can think of startupprogress as Startup,startup is the starting point. The startup process of HDFs mainly refers to the startup of Namenode. In HDFs The server-side project has a dedicated package directory to hold this part of the code. as the startupprogress name shows, the functionality of this module is not a role for HDFS start-up control, but rather a role to initiate tracking and progress reporting . Because there is not much code under the Startupprogress package in HDFs, we can do a detailed analysis. The related class file looks like this:

In the total, I have divided the related classes under this package into 3 major categories:

The first class, startupprogress start Progress Information Core class: This large class includes the startupprogress as the core of the class and related peripheral classes, Startupprogressview and Startupprogressmetrics. The former is an "instantaneous data copy" of the Startupprogress class, and the latter is the statistical count class of startupprogress.
The second class, Phase stage class: During the Namenode startup process, there are many stages, and the execution of each stage is controlled by the phase class here.
The third class, set stage step class: The set step is a finer-grained unit than the phase phase, with a single sentence that outlines one or more steps in each phase phase.

Background: Namenode Start-up process

Before learning about the HDFs startupprogress internal code, it is necessary to understand the Namenode startup process in advance. Here is a brief description of the Namenode startup process:

1.NameNode start, the first load fsimage image file, it will take more time to parse the file.
After the 2.FsImage image file is loaded, apply applies the Editlog log file to the Namenode memory, which can take a certain amount of time if the Editlog is very numerous.
3.EditLog application finished, Namenode will do a checkpoint operation. A new Fsimage image file is generated, that is, the Fsimage file is now the latest metadata.
4. After all the above file data has been loaded, Namenode will enter the SafeMode Safe mode and wait for the Datanode block to escalate. The number of reported blocks reached the threshold required by Safe mode, and the safe mode would naturally exit, and then Namenode formally started.

Stages and steps within the startupprogress

The 4 start-up processes of the previous section Namenode have a corresponding declaration in the phase class of startupprogress, with the following code:

 Public enumPhase {/** * The Namenode is loading the fsimage file into memory. */Loading_fsimage ("Loadingfsimage","Loading fsimage"),/** * The namenode is loading the edits file and applying IT operations to the * in-memory metadata. */Loading_edits ("Loadingedits","Loading edits"),/** * The Namenode is saving a new checkpoint. */Saving_checkpoint ("Savingcheckpoint","Saving checkpoint"),/** * The Namenode have entered SafeMode, awaiting block reports from data nodes. */SafeMode ("SafeMode","Safe mode"); ...

In each stage, a finer-grained step is formed. The StepType class under the Startupprogress package defines a total of the following types of step:

 Public enumStepType {/** * The Namenode has entered SafeMode and are awaiting block reports from * datanodes. */Awaiting_reported_blocks ("Awaitingreportedblocks","Awaiting reported blocks"),/** * The Namenode is performing a operation related to delegation keys. */Delegation_keys ("Delegationkeys","delegation Keys"),/** * The Namenode is performing a operation related to delegation tokens. */Delegation_tokens ("Delegationtokens","delegation Tokens"),/** * The Namenode is performing a operation related to Inodes. */Inodes ("Inodes","Inodes"),/** * The Namenode is performing a operation related to cache pools. */Cache_pools ("Cachepools","Cache Pools"),/** * The Namenode is performing a operation related to cache entries. */Cache_entries ("Cacheentries","Cache Entries"); ...

The relationship between phase and set is contained and contained. Each phase and set will undergo the following 3 state changes in total during the execution:

• 1.PENDING
• 2.RUNNING
• 3.COMPLETE

The attribution of these 3 states will be determined by the start time and end time of the phase or step . The related structure of phase and set is as follows:

During the start-up process of namenode, the execution of the above phase and step is experienced, and their respective execution status is recorded by the corresponding tracking class. Phase corresponds to the phasetracking,step corresponding to the steptracking. They all inherit from abstracttracking. The lowest-level trace class contains 2 of the most basic variable information: The start time and the end time. as follows:

abstractclass AbstractTracking implements Cloneable {  long beginTime = Long.MIN_VALUE;  long endTime = Long.MIN_VALUE;  ...}

How do you track the progress of this process inside a Class 2 tracker? First we look at the Phasetracking class:

finalclass PhaseTracking extends AbstractTracking {  String file;  long size = Long.MIN_VALUE;  // Step步骤集合以及对应的跟踪器类  final ConcurrentMap<Step, StepTracking> steps =    new ConcurrentHashMap<Step, StepTracking>();  ...

As can be seen from the above code,phase and step are indeed contained with the included relationship, phase tracking monitoring needs to rely on the tracking monitoring of each sub step .

Here is the tracking class for steptracking:

finalclass StepTracking extends AbstractTracking {  // 计数值  countnew AtomicLong();  // 总数值  long total = Long.MIN_VALUE;  ...

In this tracking class, there are 2 indicator variables: The count value and the total value. The count value variable count is deliberately designed as the Atomiclong type, which indicates that there will be a concurrency contention in the process's count update process, and that using Atomiclong can guarantee an atomic update .

The principle and invocation of startupprogress

The previous section details the structure of the phase and step, and let's look at how these objects are called during startup of Namenode. This part of the logic is in the core class startupprogress. We find the declaration of this class:

public class StartupProgress {  // 包含了各个Phase阶段  for access by StartupProgressView  final Map<Phase, PhaseTracking> phases =    new ConcurrentHashMap<Phase, PhaseTracking>();  ...

We can see in a moment that the Startupprogress class contains a collection of stages. We can assume that this phases will be in the namenode of the subsequent start-up process to join the various phase. So when was startupprogress created? The answer is divided into 3 related calls in the code shown below:

  // 1.初始化StartupProgress实例子  privatestaticfinalnew StartupProgress();

  protected void initialize(Configuration conf) throws IOException {    ...    2.注册StartupProgress Metric统计    StartupProgressMetrics.register(startupProgress);

  privatevoidstartHttpServer(finalthrows IOException {    newthis, getHttpServerBindAddress(conf));    httpServer.start();    // 3.赋值startupProgress到NameNodeHttpServer中,这个将会与StartupProgressView相关    httpServer.setStartupProgress(startupProgress);  }

When the instance is initialized, all the phase objects are added, which is the 4 stages mentioned earlier.

  publicStartupProgress() {    for (Phase phase: EnumSet.allOf(Phase.class)) {      new PhaseTracking());    }  }

So now the question is, how does the Startupprogress object mark the beginning of the Phase,set run? We take the load_fsimage phase as an example.

  private boolean loadFSImage(FSNamesystem target, StartupOption startOpt,      MetaRecoveryContext recovery)      throws IOException {    ...    // NameNode中获取StartupProgress实例    StartupProgress prog = NameNode.getStartupProgress();    // 开始LOADING_FSIMAGE阶段    prog.beginPhase(Phase.LOADING_FSIMAGE);    File phaseFile = imageFiles.get(0).getFile();    // 设置此阶段用到的文件路径以及大小    prog.setFile(Phase.LOADING_FSIMAGE, phaseFile.getAbsolutePath());    prog.setSize(Phase.LOADING_FSIMAGE, phaseFile.length());    boolean needToSave = inspector.needToSave();    ...

Continue to look in the Beginphase method

  publicvoidbeginPhase(Phase phase) {    // 判断启动过程是否结束    if (!isComplete()) {      // 如果没有结束,设置此阶段的开始时间为当前时间      phases.get(phase).beginTime = monotonicNow();    }  }

Once the stage has been executed, it will undergo internal step execution, for example, after the start of load fsimage, it will undergo inodes loading on the way, the code is as follows:

    publicvoidloadthrows IOException {      checkNotLoaded();      assertnull"curFile is null";      // 获取StartupProgress对象实例      StartupProgress prog = NameNode.getStartupProgress();      // 新建INODES的Step对象      new Step(StepType.INODES);      // 开始此Step      prog.beginStep(Phase.LOADING_FSIMAGE, step);      ...

We continue to enter the Beginstep method,

  publicvoidbeginStep(Phase phase, Step step) {    // 同样判断启动过程是否已完成    if (!isComplete()) {      // 设置开始时间      lazyInitStep(phase, step).beginTime = monotonicNow();    }  }

This part of the code is very similar to the previous Beginstep method, but the Lazyinitstep method is called here, and we enter this method to see the logic in one step closer:

  privatelazyInitStep(Phase phase, Step step) {    // 获取此阶段的Step集合    ConcurrentMap<Step, StepTracking> steps = phases.get(phase).steps;    // 如果不存在目标step,则加入    if (!steps.containsKey(step)) {      new StepTracking());    }    return steps.get(step);  }

At the end of phase and step, the corresponding Endphase/endstep method is executed, and the relevant count value is set.

prog.endPhase(Phase.LOADING_FSIMAGE);

    ...        // Step结束之后,结束此step        prog.endStep(Phase.LOADING_FSIMAGE, step);        // Now that the step is finished, set counter equal to total to adjust        for possible under-counting due to reference inodes.        // 同时设置Counter        prog.setCount(Phase.LOADING_FSIMAGE, step, numFiles);    ...

The other phase are completely similar to the step implementation process, and are not described here. But one step is special, and I would like to mention it as a step in the loading_edits phase. Each step of this stage is to apply Editlog file data into memory. But this type of step does not exist beforehand, but is directly based on the Editlog filename new, the code is as follows:

  long loadFSEdits(EditLogInputStream edits, long expectedStartingTxId,      StartupOption startOpt, MetaRecoveryContext recovery) throws IOException {    StartupProgress prog = NameNode.getStartupProgress();    // 根据edits文件输入流创建新的step    Step step = createStartupProgressStep(edits);    // 开始此step    prog.beginStep(Phase.LOADING_EDITS, step);    ...

The corresponding relationship between the phase and set in the startup process is as follows:

Startupprogress "instantaneous image": Startupprogressview

It took a certain length of time to introduce the principle and process of the internal progress of startupprogress, so what is the way to enable users to obtain this data information? The designers of HDFs have already helped us to consider such demand points. The answer is through Startupprogressview. As the title describes, Startupprogressview is an "instantaneous data copy" of the Startupprogress class. This point in time is the state data, and after 5 minutes it becomes another state data. Here we focus on the following 2 areas:

• Startupprogressview How to synchronize the data of the startupprogress boot process.
• Startupprogressview how to present its own data to external users.

First of all, Startupprogressview needs to make a copy of the current startupprogress progress data before showing the progress data externally. The relevant code is as follows:

  StartupProgressView(StartupProgress prog) {    phases = new HashMap<Phase, PhaseTracking>();    for (Map.Entry<Phase, PhaseTracking> entry: prog.phases.entrySet()) {      // 对StartupProgress中的每个Phase的跟踪数据做拷贝      phases.put(entry.getKey(), entry.getValue().clone());    }  }

in each phasetracking copy process, a steptracking copy of the data is made nearly one step. How can users access their data after the copy is complete? HDFs internally provides us with the corresponding HTTP request service, in other words, we can get Startupprogressview data through HTTP requests. This allows us to view the HDFs boot progress data directly through the Web interface. The HTTP request Service processing code is as follows:

 Public  class startupprogressservlet extends dfsservlet {...@Override  protected void Doget(HttpServletRequest req, HttpServletResponse resp)throwsIOException {Resp.setcontenttype ("Application/json; Charset=utf-8 ");//Get startupprogress instancesStartupprogress prog = Namenodehttpserver.getstartupprogressfromcontext (Getservletcontext ());//Get a copy of the data at this momentStartupprogressview view = Prog.createview ();//Return copy data in JSON formatJsongenerator JSON =NewJsonfactory (). Createjsongenerator (Resp.getwriter ());Try{Json.writestartobject ();      Json.writenumberfield (Elapsed_time, View.getelapsedtime ());      Json.writenumberfield (Percent_complete, View.getpercentcomplete ());      Json.writearrayfieldstart (phases);      ...      }      Json.writeendarray ();    Json.writeendobject (); }finally{Ioutils.cleanup (LOG, JSON); }  }

The following is a startupprogress boot progress infographic viewed through the Namenode Web interface, which is fully consistent with the previously mentioned phase and step comparisons.

The above is the article to be elaborated on the HDFs startup related content, hope to bring you harvest.

Startupprogress Start-up tracking analysis for HDFs