126th: Datanode directory metadata structure of Hadoop cluster Management learning notes

Section 126 :Hadoop Cluster Management Datanode directory meta-data structure detailed study notes

Namenode is The meta-data that manages the HDFs file system

Datanode is responsible for the management of the data on the current node, which is created automatically in the initial stage. datanode is not in format when using hdfs DFS namenode format .

In datanode , directories are stored by file information.

Datanode exist in hadoop-2.6.0/dfs/data/current on a specific node .

the version content of Datanode is similar to the version content of namenode .

Storageid: when communicating with datanode in namenode ,datanode will Namenode get storageid. For the specific datanode ,storageid is unique.

This ID is required to access namenode from datanode .

Namenode can use this attribute to distinguish between different Datanode.

Other content:clusterid/ctime/storagetype

From the point of view of data storage, There are many files in Datanode that start with blocks , which are two types of files:

The 1.hdfs itself is a block file with specific raw data.

2. Metadata for the block, suffixed by Meta .

The block file contains the original file of the stored file, and the metadata contains the section of the header and current block, and the checksum information.

When Hadoop sets the directory structure for datanode , subdirectories are created to hold the data blocks and metadata information for the file. If the current directory contains up to four blocks (set:dfs.datanode.number.blocks), the subdirectory is created.

A tree-like topology at a time. This makes it easy to manage files in each directory with Datanode, avoiding many of the block management challenges.

You can also configure Dfs.datanode.dir to point to a number of different directories in a specific configuration, and data block writes are on the same datanode node in a random-seek manner. block blocks do not repeat, unlike namenode ,namenode is a backup. Datanode is the same block backup on different machines .

above content is Liaoliang teacher Dt hadoop The first-class combat classic "section 126 Talk about the study notes.
Liaoliang: " flink docker , android technical Chinese evangelist. spark Dean and chief expert of Asia Pacific Research Institute, dt android Span style= "font-family: the song Body;" > Soft and hard integrated source-level experts, English pronunciation magician, fitness enthusiasts.

Public account:Dt_spark

Contact email [email protected]

Tel:18610086859

qq:1740415547

Number:18610086859

Sina Weibo:ilovepains

Liaoliang's first Chinese Dream: Free for the whole society to cultivate thousands of outstanding big Data practitioners!

Can be donated by Liaoliang Teacher's number 18610086859 to send red envelopes, now released Liaoliang free video complete the following:

1, "Big Data sleepless Night:Spark kernel decryption (total )":http://pan.baidu.com/s/1eQsHZAq

2, "Hadoop in- depth Combat classic" Http://pan.baidu.com/s/1mgpfRPu

3 spark Pure combat Public Welfare Forum "   http://pan.baidu.com/s/1jGpNGwu 
4 Span style= "font-family: the song Body;" >, " scala The classic of the practical,"   http://pan.baidu.com/s/1sjDWG25 
5 docker   http ://pan.baidu.com/s/1ktpl8uf 
6 spark Asia Pacific Research Institute spark   http://pan.baidu.com/s/1i30Ewsd 

7,Spark Combat Master Road All six stages video:http://edu.51cto.com/pack/view/id-144.html

8, "Big Data Spark Enterprise-level combat" purchase http://item.jd.com/11622851.html

Section 126 video website address:

51CTO

Http://edu.51cto.com/lesson/id-77867.html

126th: Datanode directory metadata structure of Hadoop cluster Management learning notes