Changes of WebUI during Namenode startup in HDFS 2

In HDFS1, the Namenode boot sequence is as follows:

1. Read Fsimage file

2. Read the edit logs file and perform the inside Operation line by row

3. Write checkpoint, generate new Fsimage (old Fsimage + editlogs)

4. Enter Safe mode and wait for Datanodes block reports until the block percent of the smallest replication number is withdrawn

During Safe mode, the client cannot modify namespace information, nor does it allow replication blocks,client is basically blocked.

And some problems lead to a long time to request requests from Namenode to client.

1. If the editlogs becomes very large (for example, because the secondary Namenode service hangs, does not merge a newer fsimage in time), causes reads the very big editlogs, carries on the operation to be relatively slow

2. General Fsimage and Editlogs files will do RAID 1 mirroring, write a new Fsimage checkpoint will write a number of copies, which requires many copies of the success of the operation after the successful, so any disk has performance bottlenecks, will lead to delays

Another problem is that the Namenode Web UI server is started after writing checkpoint, which leads to the inability of an administrator to see the entire startup progress visually through Wwebui during startup for a long time. Can only be seen through Namenode.hadoop.log.

However, a feature has been added to the 2.1.0beta to view nm startup status (https://issues.apache.org/jira/browse/HDFS-4249) on the WebUI. It does this by putting the Web UI server startup in the NM boot sequence so that users can see it as soon as possible. And on the UI, the details of the different stage are added, including the absolute path of the loaded fsimage on the NM node, its file size, the number of loaded inode, and, in safe mode, the percentage of blocks and block totals that have been received, Users can roughly estimate when to exit Safe mode. HDFs not only is the interior becoming more robust and stable, but the user experience on the periphery is becoming more and more awesome.

NM Startup Progress:

More Wonderful content: http://www.bianceng.cnhttp://www.bianceng.cn/webkf/tools/

In addition to WebUI, users can also wget http://namenode-address:50070/startupProgress to get the JSON format startup information

{"ElapsedTime": 35866, "PercentComplete": 1, "phases": [{"Name": "LOADINGF   
            Simage "," status ":" COMPLETE "," PercentComplete ": 1," ElapsedTime ": 165, "File": "/data/yarn/name/current/fsimage_0000000000000002434", "size": 22763, "St   
                    EPS ': [{' Name ': ' Inodes ', ' count ': 215,  
                "Total": 215, "PercentComplete": 1, "ElapsedTime": 25   
                    }, {"Name": "Delegationkeys", "Count": 0,  
                "Total": 0, "PercentComplete": 1, "ElapsedTime": 0   
                    }, {"Name": "Delegationtokens", "Count": 0, "Total": 0,   
                    "PercentComplete": 1, "ElapsedTime": 0}] }, {"Name": "Loadingedits", "status": "COMPLETE", "Perce Ntcomplete ": 1," ElapsedTime ": 171," steps ": [{] Co   
                    Unt ": 1," File ":"/data/yarn/name/current/edits_0000000000000002435-0000000000000002435 ",   
                    "Size": 1048576, "Total": 1, "PercentComplete": 1,   
                    "ElapsedTime": {"Count": 1044, "File": "/data/yarn/name/current/edits_0000000000000002436-0000000000000003479", "size": 1048576, "Total": 1044, "PercentComplete": 1, "elapse  
          DTime ": 155      }]}, {"Name": "Savingcheckpoint", "status": "COMP  
                Lete "," PercentComplete ": 1," ElapsedTime ":", "Steps": [ {' name ': ' Inodes ', ' count ': 299, ' file ': '/data/yarn/ Name ", Total": 299, "PercentComplete": 1, "elapsedti Me ":", {"name": "Delegationkeys", " Count ": 0," file ":"/data/yarn/name "," Total ": 0," PE  
                    Rcentcomplete ": 1," ElapsedTime ": 0}, {   
                    ' Name ': ' Delegationtokens ', ' count ': 0, ' file ': '/data/yarn/name ', "Total":0, "PercentComplete": 1, "ElapsedTime": 0} ]}, {"Name": "SafeMode", "status": "COMPLETE", "percent Complete ": 1," ElapsedTime ": 35118," steps ": [{] Na   
                    Me ":" Awaitingreportedblocks "," Count ": 218," Total ": 218,  
    "PercentComplete": 1, "ElapsedTime": 0}]} ]  
}

Author: csdn Blog Lalaguozhe