124th: The Fsimage of Hadoop cluster management and the edits working Mechanism insider detailed study notes

Section 124 :the fsimage and edits working mechanism ofHadoop Cluster Management Insider Detailed learning Notes

When a client writes a file to HDFs , it is first recorded in the edits file.

metadata is also updated when edits is modified.

The client will not see the latest information until the edits is updated on each HDFS update.

Fsimage: is a mirror of metadata in namenode , commonly called a checkpoint.

The general beginning of the operation of the Namenode in the edits , why not put in the fsimage it?

Because fsimage is the full image of Namenode , the content is very large, if each time it is loaded into memory, a tree-like topology is generated, which is very memory-consuming and CPU- .

The contents contain all the files and files in the datanode under the Namenode Management block and block where the Datanode meta-data information. As edits content grows, it needs to be merged at a certain point in time and fsimage .

Merge process:

Complete the merge isSecondarynamenode, you will be askedNamenodestop usingedits,temporarily place a new write operation in a new file (edits.new). SecondarynamenodefromNamenodethrough theHTTP GETGetedits, because to andFsimageMerge, so it is also throughHTTP GETthe way to putFsimageload into memory, then perform specific operations on the file system, withFsimageMerge to create a newFsimage, and then putFsimagesent toNamenode, throughHTTP POSTthe way. NamenodefromSecondarynamenodeobtained aFsimagewill then put the originalFsimageReplace with a newFsimage,putedits.newbecomeedits. Updates are alsoFstime.

when Hadoop enters Safe mode, administrators need to use dfsadmin 's save namespace to create new checkpoints.

Secondarynamenode Memory and Namenode required to merge edits and fsimage almost, so the general Namenode and the Secondarynamenode placed on different machines.

Fs.checkpoint.period: default is one hours (3600s)

Merge is also triggered when the fs.checkpoint.size:edits reaches a certain size (default 64MB)

The above content is a study note from the Liaoliang teacher DT Big Data Course Section 124.
DT Big Data public account:dt_spark

Liaoliang Teacher qq:1740415547

Liaoliang teacher Number:18610086859
Liaoliang's first Chinese Dream: free education for the whole society - Thousands of great data practitioners!
You can donate big data, Internet +,liaoliang, Industry 4.0 through the teacher's number 18610086859 . , micro-marketing, mobile internet and other series of free combat courses, the Liaoliang has released the complete set of free video is as follows:

1, "Big Data sleepless Night:Spark kernel decryption (total )": http://pan.baidu.com /s/1eqshzaq

2, "Hadoop in- depth Combat classic"http://pan.baidu.com/s/1mgpfRPu

3, "Spark Pure Combat Public Welfare Big Forum"Http://pan.baidu.com/s/1jGpNGwu

4, "Scala Practical classic"http://pan.baidu.com/s/1sjDWG25

5, "Docker Great Public Forum"Http://pan.baidu.com/s/1kTpL8UF

6,Spark Public Forum ofspark Asia-Pacific Institute HTTP://PAN.BAIDU.COM/S/1I30EWSD

7,DT Big Data Dream Factory Spark,Scala,Hadoop of all videos, PPT and the code in the Baidu Cloud Network disk Link:

Http://pan.baidu.com/share/home?uk=4013289088#category/type=0&qq-pf-to=pcqq.group

Liaoliang free 51CTO released in the collection of Big Data spark,hadoop, Scala , Docker Video:  

1, "Scala Beginner's introductory classic video course"http://edu.51cto.com/lesson/id-66538.html

2, "Scala advanced Advanced Classic Video Course"http://edu.51cto.com/lesson/id-67139.html

3, "Akka-in- depth Practical Classic Video Course"http://edu.51cto.com/lesson/id-77672.html

4, "Spark Asia-Pacific Research Institute wins big Data Times Public Welfare lecture"http://edu.51cto.com/lesson/id-30815.html

5, "cloud computing Docker Virtualization Public Welfare Big Forum"http://edu.51cto.com/lesson/id-61776.html

6, "Spark Lecture Hall (manual operation of Pure Combat)"http://edu.51cto.com/lesson/id-78653.html

7, "Hadoop Classic video Lessons - cluster,HDFS,Yarn , MapReduce " http://edu.51cto.com/lesson/id-77141.html

8, "a technical perspective on what Hadoop is"http://edu.51cto.com/course/course_id-1151.html

"DT Big Data Dream Factory" team first Chinese Dream: free for the community to cultivate Thousands of outstanding big data practitioners. continue to share big data, Internet +, 4.0 , and industrial in 4 every morning. , micro-marketing, mobile internet and other fields

124th Video Website Address:

51CTO

Http://edu.51cto.com/lesson/id-77860.html

124th: The Fsimage of Hadoop cluster management and the edits working Mechanism insider detailed study notes