HDFs name node start

In the Namenode class there is a main function, which is the entry function that initiates the Namenode node. A Namenode object is created in the main function of the Namenode class by Method Createnamenode (), and then the Namenode.join () function is called to wait for Namenode to stop

In the Createnamenode () function, first obtain a configuration object that is used to load the configured parameters required to start the Namenode node. Then get the run parameters. It is then processed separately according to the values of the running parameters:

If it is a format operation, it is formatted:
1. Get the name node namespace mirror fsimage saved directory and edit log edits save Directory
2. Create Fsnamesystem Object
A. First create the Fsimage object: Initialize the member variable Fseditlog object, initialize the property of the parent class storage
I. Storage type Storagetype is the name node type
II. For each storage directory: Create storagedirectory: Initialize root directory and name node directory type (image or edit directory)
Add to member variable storagedirs list
B. Initializing some parameters in the Fsnamesystem object

C. Creating a Fsdirectory object:
Assigning a Fsimage object to a member variable Fsimage
Initialize root Inodedirectorywithquota object (ROOTDIR member variable):
Initializes the parent class Inodedirectory:name to the string, permission 755,childer is empty
Initialize node quota: Nsquota Infinite
Initialize space quotas: 1
Nodes Nscount set to 1

D.fsdirectory object as properties of Fsnamesystem
3. Format the operation ( The Format method in the Fsimage Class):
A. Further initializes the member variables in the Fsimage object:
Layoutversion,namespaceid,ctime, Checkpointtime
B. Format processing for each storage directory:
1. ( Cleardirectory method) If the current subdirectory is present in the directory, clear the existing subdirectory and re-create the
in the Active Directory
2 to create a file lock In_user.lock

3. In the current directory, according to the type of storage directory (Savecurrent method, at this time only need to save the root directory information):
A. (Judging by the name node directory type) If it is a metadata image file: Create a Fsimage file, and write Layoutversion,namespaceid,
the number of nodes corresponding to the root node (initially 1), Generate time Genstamp (initially 1000L)
and then save the root node information: name Length (0), Path (""), Replica factor (0), Last modified (0), data block size (0), number of data blocks (0), node quota, space quota, user name, group name permission information
B. If you are editing a log file: Create a edits file, write Layoutversion
C. ( Storagedirectory Write method):
1. Under the current directory (root directory), check and create the subdirectory image, and in the image directory, check and create the Fsimage file, write related information, such as layoutversion, description information
2. Create the version file in the current directory and write the information layoutversion,
Storagetype,namespaceid,ctime

In the case of a finalize operation, some finalize processing occurs:
1. Get the name node namespace mirror fsimage saved directory and edit log edits save directory
2. Create a Fsnamesystem object
A. First create a Fsimage object: Initialize the member variable Fseditlog object, initialize the properties of the parent class storage
I. Storage type Storagetype is the name node type
II. For each storage directory: Create Storagedirectory: Initialize the type of root directory and name node directory (image or edit directory)
Add to member variable storagedirs list
B. Initializing some parameters in a Fsnamesystem object

C. Create a Fsdirectory object:
Assigning a Fsimage object to a member variable Fsimage
Initialize the root Inodedirectorywithquota object (ROOTDIR member variable):
Initializes the parent class Inodedirectory:name to "" String, permission 755,childer is empty
Initialize node quota: Nsquota Infinite size
Initialize Space quota: 1
Number of nodes Nscount set to 1

D.fsdirectory object as a property of Fsnamesystem
3.FINALIZE Processing:
A. Processing for each storage directory
1. See if the previous directory exists in the root directory and there is no throw exception
2. Then rename the previous directory to the Finalized.tmp directory and, when finished, delete the finalized.tmp directory

If it is a recover operation, the recovery-related action is performed:
1. Get the name node namespace mirror fsimage saved directory and edit log edits save directory
2. Create a Fsnamesystem object
A. First create a Fsimage object: Initialize the member variable Fseditlog object, initialize the properties of the parent class storage
I. Storage type Storagetype is the name node type
II. For each storage directory: Create Storagedirectory: Initialize the type of root directory and name node directory (image or edit directory)
Add to member variable storagedirs list
C. Initializing some parameters in the Fsnamesystem object

B. Create a Fsdirectory object:
Assigning a Fsimage object to a member variable Fsimage
Initialize the root Inodedirectorywithquota object:
Initialize node quota: Nsquota Infinite size
Initialize Space quota: 1
Number of nodes Nscount set to 1
Initializes the parent class Inodedirectory:name to "" String, permission 755,childer is empty
D.fsdirectory object as a property of Fsnamesystem

3.RECOVER Processing:
Repeat the first step of the formatting process above, then the second step, and then (call the Loadfsimage (Metarecoverycontext recovery) method)
A. Read the Fstime file under the current directory of each storage directory to get the last checkpoint time. Then, in these directories, obtain the appropriate directory for the maximum checkpoint, and check the directory for no need to recover from the last outage (Recoverinterruptedcheckpoint method):
1. If the Fsimage.ckpt file exists in the current directory,
Edits.new file also exists, at this point we have to delete
Fsimage.ckpt file because we don't know if Namenode crashes when Fsimage has been uploaded complete

2. If the Fsimage.ckpt file exists in the current directory,
Edits.new file does not exist, the file upload is complete at this time we only need to delete the Fsimage file file, the
FSIMAGE.CKPT file renamed to Fsimage file

3. Read the version file in the current directory: Read Layoutversion,storagetype,
Namespaceid,ctime

4. Load the Fsimage file in this directory:
A. Read Layoutversion,namespaceid, total number of nodes, generation time Genstamp
B. Iterate through each node:
1. Read path length, path name, copy factor, modification time, data block size, data block
Number, for each block of data, read the data block information:
Blockid,numbytes,generationstamp
2. If the node is a directory node, read node quotas, space quotas
3. Read user name, group name, permission information
4. For the root directory: if the node quota or the space quota is not 1, update the root directory RootDir
Quota, and update its modification time, permission information, etc.
5. Add the node to the parent node (addtoparent):
A. If the node is a directory node and the node quota or space quota is not-1, the
Create a Inodedirectorywithquota object, or create a
Inodedirectory Object
---creating Inodedirectory objects:
Initialized properties: modificationtime,accesstime, permission-related information
But uninitialized property: Children,name,parent Property Information

B. If the node is a file node, create a Inodefile object
---creating Inodefile objects:
Properties are initialized: Modificationtime,accesstime, permissions-related information,
Replication,preferredblocksize
But uninitialized property: Name,parent,blocks Property Information

(Call RootDir's Addtoparent method)
C. Set the file or directory name for the node (Name property)
D. Get the parent node and add the current node to the parent node's children collection
(The child node is added to the parent node of the children, and the child node's parent points to it)
E. Check the parent directory node modification time and the current node modification time, take a larger value as
Modify time for parent directory node
F. For file nodes, if the file has data, there is a data block. For each block of data in the file:
A. Create the Blockinfo object and place the object in the Blocksmap
Gset<block, blockinfo> blocks collection (or get Blockinfo object from blocks collection), set property inode to current file node
Initializes the properties of the parent block when the Blockinfo object is created---
The triplets Array is established (the content is not initialized), and the Inode property is initialized
B. Adding a Blockinfo object to the member variable blocks array of a file node

6. Start loading the file information that is currently being written:
A. Number of read-write files
B. Corresponding to each file being written
1. Read path length, path, copy factor, modification time, data block size, number of blocks, data block information for each block of data:
Blockid,numbytes,generationstamp
2. Read user name, group name, permission information
3. Read client name, read client machine name
4. Read the number of data nodes corresponding to the file and load the data node description information:
A. Create a Datanodedescriptor object Clientnode
B. Read the data into the properties in the object's parent class Datanodeid:
Name,storageid,infoport
C. Read the data into the properties of the parent class Datanodeinfo:
Ipcport,capacity,dfsused,remaining, etc.
5. Create the Inodefileunderconstruction object:
Initialize parent class Inodefile (IBID.)
Set the variable name (the path is saved), Clientname,clientmachine,clientnode is null
6. Get the file path and find the old file node under RootDir
7. Replace the old file node
A. Removing the old file node from the parent node: Removes the current node from the Childer collection.
Parent points to NULL
B. Add the current node to the parent node's children collection, and the parent of the current node points
C. For each block of data in the file:
1. Create the Blockinfo object and place the object in the
Gset<block, blockinfo> blocks collection (or get Blockinfo object from blocks collection), set property inode to current file node
Initializes the properties of the parent block when the Blockinfo object is created---
The triplets Array is established (the content is not initialized), and the Inode property is initialized
2. Add the Blockinfo object to the member variable blocks array of the file node
7. (Addlease method)
Create lease Lease object: Holder client name, lastupdate to current time
Add to Leases collection: Key client name, value lease itself
Add to Treeset<lease> sortedleases
Add to Sortedleasesbypath Collection: Key path name, value lease itself
Add a path to the lease corresponding to the Paths collection

5. Load the Edit log:
A. Open the edits file and read the log version number
b. Get the opcode
1. If you are creating a file operation or closing a file operation:
A. Path to read files, number of replicas, modification time, access time, block size, data Block list, permissions (user name, user group name, and access rights)
If you are creating a file operation, also read the client name, the host on which the client resides
B. For a file close operation, remove the current node from the parent node
C. Calculate node quotas and space quotas for the old node of the file:
Node Quota: Nscount =1
Spatial mates: Sum the size of each chunk, if the last chunk is in writing, use the recommended block size
D. All parent directories of the old node recalculate node quotas and space quotas (minus quotas for the current node)
E. Setting the modified time of the parent directory to the modified time that has been read
F. Delete the data blocks owned by the old node file:
The parent is set to null
The Inode object corresponding to each block is set to NULL
Blocks set to NULL
G. Delete from the Sortedleasesbypath collection according to the file path,
Delete the dependent path in the lease and remove the lease from the leases and sortedleases collections if the lease is empty
H. Adding a new node for the current file, creating a new file node
I. Compute node quotas and space quotas for the old node of the file:
Node Quota: Nscount =1
Spatial mates: Sum the size of each chunk, if the last chunk is in writing, use the recommended block size
J. Setting the name of the new node as the file path
K. All parent directories of the new node recalculate node quotas and space quotas (plus quotas for the current node)
H. The current node is added to the parent node corresponding to the Childer binding, and the parent of the current node points

I. For each block of data in the file:
A.gset<block, blockinfo> blocks collection, gets the corresponding Blockinfo object
B. Adding a Blockinfo object to the member variable blocks array of a node

J. If you are creating a file operation, create a new Inodefileunderconstruction object
Replace the original node with inodefileunderconstruction
Create lease Lease object: Holder client name, lastupdate lease update time
Add to Leases collection: Key client name, value lease itself
Add to Treeset<lease> sortedleases
Add to Sortedleasesbypath Collection: Key path name, value lease itself
Add a path to the lease corresponding to the Paths collection

6. Save the contents to Fsiamge (Savenamespace method):
1. Turn off the original edit log output stream
2. Set the properties of the Fsimage object Checkpointtime to the current time
3. For each storage directory:
1. Rename the current directory into the Lastcheckpoint.tmp directory
2. Create a current directory
3. Call the Savecurrent method to save the directory:
1. Under the current directory, separate processing according to the type of storage directory:
A. If it is a metadata image file:
1. Create the Fsimage file and write to Layoutversion,namespaceid,
Number of nodes, generation time Genstamp
2. Then save the root node information: name Length (0), Path (""), copy factor (0), last modification time (0), data block size (0), number of data blocks (0), node quota, space quota, user name, group name permission information
3. Save the remaining node information:
A. Save all child nodes of the current node:
For each child node, save the name length, path:
1. If the child node is a file node, save the copy factor, modify the time,
Access time, specified data block size, number of data blocks, and data block information, last saved user name, group name permission information
For directory nodes: Save replica Factor (0), modify
Time, Access time (0), data block size (0), number of data blocks (-1),
Node quotas, space quotas, user names, group name permission information

If the child node is a directory node, save its child nodes recursively
4. Save the file information that is being built:
A. Number of total Files saved (Leasemanager.countpath ())
B. Read each file path from the Sortedleases , in addition to saving some file-related information, but also save the client name and client host name, save 0
Create a name node:
1. Create a Fsnamesystem object:
A. Configure some parameter information
B. Create a Fsdirectory object and assign it to the property dir:
1. First create the Fsimage object: Initialize the member variable Fseditlog object, initialize the property of the parent class storage, assign the Fsimage object to the member variable in fsdirectory fsimage
I. Storage type Storagetype is the name node type
2. Initialize root Inodedirectorywithquota object:
Initialize node quota: Nsquota Infinite size
Initialize space quota: 1
Number of nodes Nscount set to 1
initializes the parent class Inodedirectory:name to the string, permission 755,childer is empty
C. Gets the startup item
D. If the startup item is formatted:
1. For each storage directory: Create Storagedirectory: Initializes the root directory and the name node directory type (either image or edit directory) to the member variable storagedirs list
2. Format processing (abbreviated)
3. The startup entry is set to regular
E. For each storage directory, analyze the storage state of the directory and do the related recovery processing
G. Judging by the startup item
1. If it is an upgrade operation:
A. Load the image file of the latest checkpoint first, load the corresponding log (loadfsimage (recovery) method)
B. Set the CTime of the Fsimage object to the current time, layoutversion,checkpointtime to the current time

For each storage directory:
C. Rename the current directory into the previous.tmp directory
D. Call savecurrent method to save content to a new directory
E. Rename the Previous.tmp directory to the previous directory
2. If it is an import operation, do import
3. If it is a rollback operation:
A. Create a Fsimage object: Initialize the member variable Fseditlog object, initialize the property of the parent class storage, assign the Fsimage object to the member variable in fsdirectory fsimage
I. Storage type Storagetype is the name node type
B. Sets the value of layoutversion for each storage directory:
reads the version file in the previous directory:
assigns the variable layoutversion, Storagetype,namespaceid,ctime and verify
C. Rename the current directory to removed.tmp
D. Rename the previous directory into the current directory
E. Delete the removed.tmp directory
F. For the rollback operation after the end or regular action:
load the latest image file, load the corresponding log

G. Call the Savenamespace method to save the Fsimage file

2. Create a Safe Mode management Object Safemodeinfo:
blocksafe:0
threshold:0.95
1. Get the total number of data blocks: Blocks.size () in the Blocksmap object
2. Loop through the Sortedleases collection in the lease manager, get the file node for each path in the lease, get the last block of data for the file node if the size is 0, the total chunk minus 1
3. The number of final secure blocks, assigned to the Blocktotal property
4. Check the security mode status:
A. See if blocksafe/blocktotal is less than 0.95 (minimum replica level is met), and if so, enter Safe mode
Reached set to 0 (enter Safe mode), laststatusreport= current time, return
B. If blocktotal is 0 or blocksafe/blocktotal is greater than or equal to 0.95 if reached<0 (indicates that Safe mode is off)
or extension <= 0 or threshold <= 0---Don't wait
Leave Safe Mode (reached =-1) and return
C. If reached>0 indicates that Safe mode is turned on, return
D.reached = Current time indicates start Safe mode
Monitor thread startup After creating Safe mode:
Startup of Safe Mode:
1. Determine if the minimum copy level is met, then continue to safe mode
2. Otherwise leave safe mode, reached = 1

3. Create and start the heartbeat check process:
A. Check your heartbeat every 5 minutes:
1. No heartbeat check in Safe mode
2. Loop through the Heartbeats collection for the Datanodedescriptor object and get the first dead node:
A. Check if the data node is a dead node:
Is the last check time lastupdate+ Heartbeatexpireinterval is less than the current time, the description is dead node
3. Synchronize Heartbeats collections and Datanodemap collections:
B. Get the Storageid of dead node and get the Datanodedescriptor object from the Datanodemap collection.
Holds all Storageid and corresponding Datanodedescriptor objects---the Datanodemap collection
Check again if the object is not reporting a heartbeat for a long time, and if so,
Delete the node:
1. Recalculate the spatial information of the Fsnamesystem object:
Capacitytotal minus the capicty of the node,
Capacityused,capacityremaining,totalload
2. Remove the node from the Heartbeats collection
3. The IsAlive of the node is set to False
4. For each block of data for the Datanodedescriptor object:
A. In the Blocksmap:
Handling of the Blockinfo object:
1. Get the Blockinfo object from the blocks collection based on the data block
2. Gets the index position of the corresponding Datanodedescriptor object from the triplets array of the Blockinfo object
3. Get the previous and last Blockinfo objects of the data node by indexing the Triplets array in the Blockinfo object
4. In the triplets array of the Blockinfo object, set the first and last of the current Blockinfo objects to be empty
5. For the previous Blockinfo object, locate the index where the corresponding Datanodedescriptor object is located in the Triplets array of the object, and point to the next Blockinfo object
6. For the latter Blockinfo object, locate the index where the corresponding Datanodedescriptor object is located in the Triplets array of the object, and point previous to the previous Blockinfo object
7. The number of elements of the Blockinfo object is reduced by one, and the position of the last data node is moved to the index where the Datanodedescriptor object is located
8. If there are no elements in the array of the Blockinfo object, you can delete the corresponding data block from the blocks
Data Block backup operation:
Get the Blockinfo object from the data block in 1.blocks and get the corresponding file node object
2. If Safe mode:
A. (using the Countnodes method) iterates over the triplets array corresponding to the Blockinfo object for that data block, obtaining the corresponding
Datanodedescriptor Object
1. Get the Data node collection corresponding to the corrupted data block
2. Iterate through each data node:
A. If the node is in the Nodescorrupt collection, the number of corrupted replicas +1:corrupt++
B. If the current data node is being revoked or has been revoked,
The number of replicas on the undo node: count++
C. Otherwise, gets the extra copy of the current node, plus 1:excess++ if the data block is more than the replica
Otherwise normal copy plus 1:live++
---excessreplicatemap is the storageid of the key data node, and the value is more than the data block collection
D. New Numberreplicas Object
B. Get the normal number of replicas, if the number of normal copies reaches SafeReplication-1, then
blocksafe--, check the security mode status

-----The following non-safe mode
C. Use the Countnodes method again to calculate the status of the replica
D. Getting the expected number of replicas
E. Priority for updating data blocks:
1. Get the old backup count and the expected number of replicas
2. Calculate the priority of the current data block:
A. 3 (do not back up) if the current backup is already greater than the expected copy
B. If the current backup is 0 and the number of cancellations is greater than 0, 0 (the most
High priority), otherwise return 2
C. If the current backup is 1, return 0
D. If the current backup is less than the expected copy, return 1
E. Otherwise returns 2
3. Calculate the priority of the data block before deleting the data node
4. If the old priority is not 3, and the pre-and post-priority changes:
Delete the corresponding data block in the priorityqueues.
5. The current priority is not 3, add the chunk to Priorityqueues
6. In Excessreplicatemap, the data node gets all the more data blocks excessblocks, and the block is removed from the Excessblocks
---excessreplicatemap key Storageid, a collection of data blocks that correspond to values
excessblockscount--
7. Get all Datanodedescriptor objects corresponding to the data block from the Corruptreplicasmap and delete the corresponding data node objects
A collection of Datanodedescriptor objects that correspond to the CORRUPTREPLICASMAP key data block and the value data block---
5. For the Datanodedescriptor object:
A. Clear the statistics for the node object: Capacity,dfsused,remaining and so on are 0
B. (recentinvalidatesets invalid copy of the data block, i.e. a copy of the waiting block)
----Recentinvalidatesets Store key Storageid, Value data block collection
Delete all invalid data blocks for the deleted node in recentinvalidatesets and recalculate the number of copies of data blocks waiting to be deleted

C. Delete the corresponding data node topology information in Clustermap:
6. Check the safety mode

4. Create and start the Lease Manager monitor process:
1. Obtain the longest lease object from sortedleases to determine whether the Premier League
2. If yes, gets all the file paths for the lease, recursively reads each file path:
A. Getting a file node object based on a file path
B. If the file does not own any data blocks:
1. Obtain the corresponding lease from the leases based on the client name:
Removelease Method:
Delete path name and lease correspondence in A.sortedleasesbypath
B. Delete the corresponding file path in the paths corresponding to the lease
C. If the lease corresponds to paths, you can delete the lease in leases
and delete in sortedleases

2. Create a new node file object
3. Replace the old node object with a new
4. Logging information (slightly)
C. If the last data block corresponding to the file is empty:
1. Get the second-to-last block of data
2. Get the Blockinfo object from the blocks from the data block and get the data node data for the object
3. Get all data nodes corresponding to the data block
4. Reset targets to these data nodes
5. Iterate through these data nodes, and if the data node IsAlive is true, the node sets the master data node:
A. Creating a Blocktargetpair object and inserting it into the BLOCKQ queue sink
B. The client that sets the file node is named Nn_recovery
C. In the lease manager, call the Removelease method to delete the lease
D. Re-add the lease:
1. Create a lease with Nn_recovery for the client name (if the lease does not exist, the lease is available based on the client name)
2. Add to leases collection and sortedleases
3. Path name and lease relationship added to Sortedleasesbypath
4. The path name is placed in the paths of the lease
D. If the lease recovery is unsuccessful:
A. Calling Removelease directly to delete a lease

5. Data block recovery/deletion thread:
1. If the block backup and delete operation is not performed in safe mode
2. Calculate the maximum number of data blocks to replicate blockstoprocess
2. Copy operation:
A. If neededreplications (the copy waiting for assignment) is empty, do not process
B. Otherwise go to blockstoprocess and neededreplications.size () smaller values as the number of blocks to be processed
C. Iterate to get data blocks and precedence in the neededreplications, which are placed in the Blockstoreplicate collection
D. For each block, the corresponding file node object is obtained according to the data block, and if the file node is empty or in construction, the corresponding data block is removed from the Neededreplications collection
E. Get the number of data block backups that the node corresponds to
F. Get the Data source node (Choosesourcedatanode method):
1. All data nodes can be obtained from the Blocksmap object
2. Get the broken data node from Corruptreplicas
3. For each data node, if the node is corrupted or the number of data blocks that the node needs to be assigned
Or if the node has been revoked or if the data block on the current node is more than that, the data node cannot be a source node
4. If the data node is being revoked, take the node as the source node, or go to a bit source data node
G. If the number of valid data blocks plus the number of copies to be copied is greater than or equal to the expected block backup,
The data block is removed from the neededreplications
H. Selecting a Target node
I. Get the corresponding file node object according to the data block, if the file node is empty or is being built, delete the corresponding data block from the Neededreplications collection
J. If the number of valid data blocks plus the number of copies to be copied is greater than or equal to the expected block backup,
The data block is removed from the neededreplications
K. Adding data blocks to the list to be backed up: replicateblocks
L. from the Pendingreplications collection, get the corresponding Pendingblockinfo object based on the data block
If the return is empty:
Create a Pendingblockinfo object (numreplicasinprogress set to its number of backups), put in the collection numreplicasinprogress+= number of backups
Otherwise numreplicasinprogress+= the number of backups, timestamp is set to the current time
M. Delete the data block from Neededreplications if the number of valid backups plus the current number of backups is greater than the data that needs to be backed up
3. Block Delete operation:
a. Calculating the smaller values of recentinvalidatesets.size () and nodestoprocess as the number of data nodes to process a copy of a block of data
B. For these data nodes:
1. Safe mode does not handle
2. Get Nodeid corresponding data node from Datanodemap
3. Get all copies of invalid data blocks from recentinvalidatesets based on Nodeid
4. Add these invalid data blocks to the corresponding properties of the data node in the Invalidateblocks collection

4. If the Timedoutitems in Pendingreplications is not empty, some replication requests are timed out,
A. Obtain these data blocks, re-calculate the number of valid backups for the data block, and put the backup request back into the Neededreplications

6. Create revocation management, create and start data node revocation check thread:
1. Set check interval recheckinterval:30 seconds
Number of data nodes per check Numnodespercheck:5
2. Start the process:
A.

7. Create the Safe Mode Management Object Safemodeinfo, and follow the new member variable blocktotal, after the call ends, the name node
Generally in safe mode
8. Create and enable the heartbeat check process:
The heartbeat check is done every 5 minutes by default: If a data node fails, remove the node from the heartbeat queue
9. Create and enable lease check thread, block copy/delete thread
10. Read the Include file and the exclude file of the data node to create and start the data node revocation check thread
11. Create the Dnstoswitchmapping object required for the network layout, and, if possible, analyze the host locations contained in the include file

HDFs name node start