HDFs metadata Parsing

1, metadata (Metadata): Maintenance of HDFs file System file and directory information, divided into memory metadata and metadata file two kinds. Namenode maintains the entire metadata. HDFs implementation, the method of periodically exporting metadata is not adopted, but the backup mechanism of metadata mirroring file (fsimage) + day file (edits) is adopted. 2, Block: The contents of the file. Path-seeking process: &http://www.aliyun.com/zixun/aggregation/37954.html >nbsp;         path information                         bocks[]                                   triplets []          client------------"INode---------------------" Blockinfo----------------------- ---"DataNode.  inode: Basic elements of files: files and directories blockinfo: File content objects Datanodedescriptor: specific storage objects. 3, Fsimage and edits checkpoint. Fsimage has 2 states, respectively, Fsimage and FSIMAGE.CKPT, which indicate that the checkpoint process will be modified to fsimage file after uploading, and the same edits has two states, edits and edits.new. 4, Namenode format scenario analysis: Traversing the metadata storage directory, prompting the user is formatted? (Namenode.java Format function) private static Boolean format (revisit conf, Boolean isconfirmationneeded) throws IOException {Collection<uri > DirstOFormat = Fsnamesystem. Getnamespacedirs (conf); Collection<uri > Editdirstoformat = Fsnamesystem. Getnamespaceeditsdirs (conf); For (iterator< uri> it = Dirstoformat.iterator (); it. Hasnext ();) {File CurDir = new file (it. Next (). GetPath ()); if (! CurDir. Exists ()) re-enters; if (isconfirmationneeded) {System. Err. Prin T ("Re-format filesystem in" + CurDir + "?) (Y or N) "); if (! (System. In. Read () = = ' Y ')) {System. Err. println ("Format aborted in" + CurDir); return true; while (System. Read ()!= ' \ n '); Discard the Enter-key} fsnamesystem ansys = new Fsnamesystem (new Fsimage (Dirstoformat, Editdirstoformat), conf); Nsys.dir.fsImage. Format (); return false; Create metadata memory mirrors, including class Fsnamesystem instantiation objects, class Fsdirectory instantiate objects, Class Fsimage objects, class edits objects. Create Fsnamesystem objects primarily complete: blockmanager,fsdirectory objects and initialize member variables. The Fsimage object mainly completes to Layoutversion, Namespaceid,ctime assigns the value to 0, the instantiation fseditlog. In class Fsdirectory, the HDFs root node RootDir is created. Fsnamesystem (Fsimage fsimage, revisit conf) throws Ioexception {this. blockmanager = new Blockmanager (this, conf); Setconfigurationparameters (conf); this. Dir = new Fsdirectory ( Fsimage, this, conf); dt618.html ">secretmanager = Createdelegationtokensecretmanager (conf); } fsimage (collection< uri> fsdirs, collection< uri> fseditsdirs) throws IOException {this (); Setstoragedirectories (Fsdirs, fseditsdirs); } void Setstoragedirectories (Collection <uri > Fsnamedirs, collection< uri> fseditsdirs) throws IOException { This. Storagedirs = new ArrayList <storagedirectory > (); This. Removedstoragedirs = new ArrayList <storagedirectory > (); ADD all name dirs with appropriate namenodedirtype for (URI dirname:fsnamedirs) {checkschemeconsistency (dirname); Boolean isalsoedits = false; for (URI editsdirname:fseditsdirs) {if (editsdirname. CompareTo (dirname) = = 0) {isalsoedits = true; fseditsdirs. Remove ( Editsdirname); Break; } namenodedirtype Dirtype = (isalsoedits)? NamenodedirtyPE. Image_and_edits:namenodedirtype. IMAGE; Add to the "storage directories, only if the"//URI is of type file://if (dirname. Getscheme (). CompareTo (journaltype.f ILE. Name (). toLowerCase ()) = = 0) {this.addstoragedir (new storagedirectory (New File (dirname. GetPath ()), Dirtype));} Add edits dirs if tightly are different from name dirs for (URI dirname:fseditsdirs) {checkschemeconsistency (dirname);//Add To the "List of storage directories, only if the"//URI is of type file://if (dirname. Getscheme (). CompareTo (Journaltype.file. Name (). toLowerCase ()) = = 0 This.addstoragedir (new Storagedirectory (dirname. GetPath ()), Namenodedirtype. edits)); Initializes the data structure in memory-mirrored data: The main fsimage Format function is complete, layoutversion: The version of the software. Namespaceid: When the format is generated, when data node registers with name node, it obtains the namespaceid of the Namenode and acts as an identity for subsequent communication with Namenode. No communication is denied for the unknown identity data node,namenode. CTime: The time that fsimage produces. Checkpointtime: Represents the time when namespace first checkpoint.   public void Format () throws IOException {thiS. layoutversion = fsconstants. Layout_version; This. Namespaceid = Newnamespaceid (); This. CTime = 0L; This. Checkpointtime = Fsnamesystem. Now (); For (iterator <storagedirectory > it = Diriterator (); it. Hasnext ();) {Storagedirectory SD = it. Next (); format (SD);} Writes a memory image to the metadata backup directory. The Fsimage format method traverses all directories for backup. If it is a fsimage file directory, call Savefsimage save Fsimage, and if it is edits, call Editlog.createeditlogfile and finally invoke the Sd.write method to create the Fstime and version files. The version file is usually last written. void format (Storagedirectory sd) throws IOException {sd.cleardirectory ();//Create currrent dir sd.lock (); try { Savecurrent (SD); finally {sd. Unlock ()} Log.info ("Storage directory" + SD. Getroot () + "super-delegates been successfully formatted."); Finally, analysis of the application of metadata scenarios: 1, format. 2. When Hadoop starts up. 3, metadata update operation. 4. If Namenode is used in conjunction with secondary Namenode, Backup node, or checkpoint node, a checkpoint operation is performed.