Hadoop:org.apache.hadoop.hdfs.server.namenode functions and roles of various classes

Take hadoop0.21 as an example.

Namenode.java: The main maintenance file system namespace and file metadata, the following is a description of the code.

/**********************************************************&http://www.aliyun.com/zixun/aggregation/37954.html ">nbsp;

* Namenode serves as both directory namespace manager and

* Inode table for the Hadoop DFS. There is a single namenode

* Running in any DFS deployment. (OK, except when there

* is a second backup/failover namenode.)

*

* The Namenode controls nonblank unacknowledged tables:

* 1) FileName->blocksequence (namespace)

* 2) Block->machinelist ("Inodes")

*

* The stored on disk and is very precious.

* The second table is rebuilt every time the Namenode comes

* Up.

*

* ' Namenode ' refers to both this class as the ' Namenode server '.

* the ' Fsnamesystem ' class actually performs most of the filesystem

* Management. The majority of the ' Namenode ' class itself is concerned

* With exposing the IPC interface and the HTTP server to the Òoutsideó world,

* Plus some revisit management.

*

* Namenode implements the ClientProtocol interface, abound allows

* Clients to ask for DFS services. ClientProtocol is not

* Designed for direct with authors of DFS client code. End-users

* Should instead use the Org.apache.nutch.hadoop.fs.FileSystem class.

*

* Namenode also implements the Datanodeprotocol interface, used by

* DataNode programs that actually store DFS data blocks. These

* methods are invoked repeatedly and automatically by the

* Datanodes in a DFS deployment.

*

* Namenode also implements the Namenodeprotocol interface, used by

* Secondary namenodes or rebalancing processes to get partial ' s

* State, for example partial blocksmap etc.

**********************************************************/

Fsnamesystem.java: The main maintenance of several tables of information: the maintenance of the file name and block list mapping relationship between the valid block set, block and Node list mapping relationship, node and block list mapping relationship; update heatbeat node LRU Cache

/***************************************************

* Fsnamesystem does the actual bookkeeping work for the

* DataNode.

*

* It tracks several important tables.

*

* 1) Valid FSName--> blocklist (kept on disk, logged)

* 2) Set of all valid blocks (inverted #1)

* 3) block--> machinelist (kept in memory, rebuilt dynamically from reports)

* 4) Machine--> blocklist (inverted #2)

* 5) LRU cache of Updated-heartbeat Rogue

***************************************************/

INode.java:HDFS abstracts files and file directories into Inode.

/**

* We keep an as representation of the File/block hierarchy.

* This is an base INode class containing common fields for file and

* Directory inodes.

*/

Fsimage.java: The inode information needs to be persisted to disk fsimage.

/**

* Fsimage handles checkpointing and logging of the namespace edits.

*

*/

Fseditlog.java: Write edits file

/**

* Fseditlog maintains a log of the namespace modifications.

*

*/

BlockInfo.java:INode is mainly the file and directory information, and for the contents of the file, this is described by block. We assume that the length of a file is size, then start with the file's 0 offset, and then divide and number the files in order, in a fixed size, and divide each chunk into a block.

/**

* Internal class for block metadata.

*/

Datanodedescriptor.java: Represents the specific storage object.

/**************************************************

* Datanodedescriptor Tracks stats on a given DataNode,

* such as available storage capacity, last update, etc.,

* and maintains a set of blocks stored on the datanode.

*

* This data businessesflat-out is a data businessesflat-out the IS internal

* to the Namenode. It is *not* sent Over-the-wire to the Client

* or the datnodes. Neither is it stored persistently in the

* Fsimage.

**************************************************/

Fsdirectory.java: Represents all directory and structure properties in HDFs

/*************************************************

* Fsdirectory Stores the FileSystem directory state.

* It handles writing/loading values to disk, and logging

* Changes as we go.

*

* It keeps the filename->blockset mapping always-current

* and logged to disk.

*

*************************************************/

Editlogoutputstream.java: All logging is through the Editlogoutputstream output, in the concrete instantiation, This group of Editlogoutputstream contains multiple editlogfileoutputstream and a Editlogbackupoutputstream

/**

* A generic abstract class to support journaling of edits logs into

* A persistent storage.

*/

Editlogfileoutputstream.java: Writes log records to edits or edits.new.

/**

* An implementation of the abstract class {@link Editlogoutputstream}, abound

* Stores edits in a local file.

*/

Editlogbackupoutputstream.java: Sends logs over the network to the Backupnode.

/**

* An implementation of the abstract class {@link Editlogoutputstream},

* Abound streams edits to a backup node.

*

* @see org.apache.hadoop.hdfs.server.protocol.namenodeprotocol#journal

* (Org.apache.hadoop.hdfs.server.protocol.NamenodeRegistration,

* int, int, byte])

*/

Backup of BackupNode.java:name node: Upgrade phase: Secondary name node-Checkpoint node (periodically save metadata, periodic Checkpoint)-"Backup Node" ( Keep a mirror in memory that is exactly the same as name Node, and update the metadata as it changes, using its own mirroring to checkpoint, without downloading from Namenode-Standby Node (hot standby)

/**

* Backupnode.

* <p>

* Backup node can play nonblank roles.

* <ol>

* <li>{@link Namenoderole#checkpoint} node periodically creates checkpoints,

* That's downloads image and edits from the active node, merges them, and

* Uploads the new image back to the active. </li>

* <li>{@link Namenoderole#backup} node keeps its namespace into sync with the

* Active node, and periodically creates checkpoints by stencils saving the

* Namespace image to Local disk (s) .</li>

* </ol>

*/

Backupstorage.java: Create the Jspool in the Backup node directory and create the edits.new to point the output stream to Edits.new

/**

* Load checkpoint from local files only if the memory the ' is empty.<br>

* Set New checkpoint Time received from the Name-node. <br>

* Move <code>lastcheckpoint.tmp </code> to <code>previous.checkpoint</code>.

* @throws IOException

*/

Transferfsimage.java: Responsible for files from name node.

/**

* This class provides fetching a specified file from the Namenode.

*/

Getimageservlet.java: is a httpservlet subclass that handles doget requests.

/**

* This class was used in Namesystem ' s jetty to retrieve a file.

* Typically used by the secondary namenode to retrieve image and

* Edit file for periodic checkpointing.

*/