Deep analysis of HDFs

This article used to view the Hadoop source, about the Hadoop source code import http://www.aliyun.com/zixun/aggregation/13428.html ">eclipse way See the first phase

HDFs Background Introduction

As the amount of data is increasing, the scope of an operating system is not enough, then allocated to more operating system management disk, but not convenient management and maintenance, the urgent need for a system to manage the files on multiple machines, this is the Distributed file Management system.

The academic definition is that a distributed file system is a system that allows files to be shared across multiple hosts over a network, allowing multiple users on multiple machines to share files and storage space. A lot of distributed file Management system, Hdfshdfs is just one of them. For a single write, multiple queries, not supporting concurrent write, small file is not appropriate. Because small files also occupy a block, the more small files (1000 1k files) block more, namenode pressure.

The basic concepts of

Ii. HDFs

The files we upload through the Hadoop shell are stored in the Datanode block, and the file is not visible through the Linux shell. Can be described in a sentence HDFs: The client's large files stored in a number of nodes in the data block. Here, there are three keywords: files, nodes, blocks of data. HDFs is around the three key words designed, we also have to grasp these three key words to learn.

The NameNode1 of the basic structure of

HDFs. function

The function of Namenode is to manage file directory structure, accept user's operation request, and manage data node. The name node maintains two sets of data, one is the relationship between the file directory and the data block, the other is the relationship between the data block and the node. The previous set of data is static, is stored on disk, through fsimage and edits files to maintain, the latter set of data is dynamic, not persisted to disk, whenever the cluster starts, will automatically build this information, so generally put in memory.

So he is the management node of the entire file system. It maintains the file directory tree for the entire file system, the file/directory meta information, and the block list for each file. Receives the user's action request.

The files include:

①fsimage (File system Mirroring): Metadata mirroring file. Stores Namenode memory metadata information for a period of time.

②edits: Operation log file.

③fstime: Save the last checkpoint time

These files are stored in the Linux file system

2. Features

<1> is a file system that allows files to be shared across multiple hosts over a network, allowing multiple users on multiple machines to share files and storage space.

<2> permeability. The act of allowing a file to actually be accessed over a network, as the program and the user sees it, is like accessing a local disk.

<3> fault tolerance. Even if some of the nodes are offline in the system, the system can continue to operate without loss of data overall.

<4> applies to a single write, multiple queries, and does not support concurrent write, small file is not appropriate

3. Directory structure &lt;1&gt; since Namenode maintains so much information, where is this information stored?

In the Hadoop source code, there is a file called Hdfs-default.xml, as shown in Figure 3.1.

Figure 3.1

&lt;2&gt; Open this file

In lines 149th and 158th, there are two configuration information, one is Dfs.name.dir, and the other is Dfs.name.edits.dir. These two files represent the location of the Namenode core files Fsimage and edits, as shown in Figure 3.2.

Figure 3.2

The value of the corresponding configuration is ${}, which is the representation of the variable, and the ER expression reads the value of the variable when the program reads the file. Then, the value of the variable Hadoop.tmp.dir in line 150th (that is, the Hadoop temporary storage path), as shown in Figure 3.3.

Figure 3.3

However, in the configuration file Core-site.xml in the previous chapter, the configured value is/usr/local/hadoop/tmp.

&lt;3&gt; We can access the Linux file system

Execute command cd/usr/local/hadoop/conf,more core-site.xml see the content shown in Figure 3.3.

Figure 3.4

As you can see, the two files are stored in the/usr/local/hadoop/tmp/dfs/name directory of the Linux file system.

&lt;4&gt; We enter this catalogue

Look at the contents of this directory, as shown in Figure 3.5.

Figure 3.5

As can be seen from the figure, Namenode's core files Fsimage and edits are stored in the current directory, while there is a file under the name directory In_ Use.lock while viewing its contents, found that the content is empty, that is, only one namenode process can access the directory, readers can try it on their own, when Hadoop is not open, the directory is not file In_use.lock, The file will not be generated until Hadoop is started.

&lt;5&gt; file Fsimage is Namenode's core document

This file is very important, lost, namenode can not be used, then how to prevent the loss of the file caused undesirable consequences. I can take a look at the code in Hdfs-default.xml again, as shown in Figure 3.6.

Figure 3.6

It is known from the description that this variable determines where the DFS namenode nametable (fsimage) should be stored on the local file system. If this is a comma-delimited list of directories, then NameTable will be replicated to all directories to be redundant (backed up to secure data). such as ${hadoop.tmp.dir}/dfs/name,~/name2,~/name3,~/name4. Then Fsimage will be copied to the ~/name1,~/name2,~/name3,~/name4 directory separately. So these directories are generally in different machines, different disks, different folders, in short, the more dispersed the better, so as to ensure the security of data. Some people will ask how to achieve on multiple machines? In fact, there is NFS file sharing system in Linux, this is not done in detail.

&lt;6&gt; take a look at edits's description

Look at a piece of code in Hdfs-default.xml as shown in Figure 3.7

Figure 3.7

It is known from the description that the variable determines the location of the Dfsnamenode storage transaction file (edits) on the local file system. If this is a comma-separated list of directories, the transaction file is duplicated in all directories for redundancy. The default value is Dfs.name.dir. (Edit save Transaction process)

the basic structure of the HDFs DataNode1. Role: The role of Datanode is to actually store data in HDFs. 2. Block

<1> If a file is very large, such as 100GB, how do you store it in Datanode? Datanode reads and writes data in blocks when it is stored. Block is the basic unit of HDFs read and write data.

<2> Suppose the file size is 100GB, starting at byte position 0, dividing each 64MB byte into a block, and so on, so that you can divide a lot of blocks. Each block is a 64MB size.

2.1 Let's take a look at the Org.apache.hadoop.hdfs.protocol.Block class,

.

Here are a few attributes, as shown in Figure 4.1.

Figure 4.1

As the above illustration shows, none of the properties in a class can store data. So block is essentially a logical concept, which means that blocks do not really store data, just partition files.

2.2 Why must be divided into 64MB size?

Since this is set in the default configuration file, we view the Core-default.xml file as shown in Figure 4.2.

Figure 4.2

The parameter Ds.block.name in the above figure refers to the block size, which is 67 108 864 bytes and can be converted to 64MB. If we do not want to use the 64MB size, we can overwrite the value in Core-site.xml. Note that the unit is byte.

2.3 Copy

<1> copy is backup for security purposes. It is because the cluster environment is unreliable that the replica mechanism is used to secure the data.

The disadvantage of a <2> replica is that it takes up a lot of storage space. The more copies you have, the more space you occupy. Compared to the risk of data loss, the cost of storage space is worth it.

<3> So how many copies of a file are appropriate? We looked at the Hdfs-default.xml file, as shown in Figure 4.3.

Figure 4.3

As you can see from Figure 4.3, the default number of replicas is 3. means that there are 3 copies of each block of data in the HDFs. Of course, every effort will be made to allocate to different Datanode servers. Imagine: If the backup of the 3 data are on the same server, then the server is down, is not all the data are lost AH?

3. Directory Structure 3.1 Datanode is a block to partition the file

So where exactly is the divided file stored? We look at the file Core-default.xml, as shown in Figure 4.4.

Figure 4.4

The value of the parameter Dfs.data.dir is where the block resides in the Linux file system. The value of the variable Hadoop.tmp.dir is described earlier, is/usr/local/hadoop/tmp, then the Dfs.data.dir full path is/usr/local/hadoop/tmp/dfs/data. Viewed through the Linux command, the results are shown in Figure 4.5.

3.2 Upload a file

We first click Pietty to open another Linux terminal, upload a file jdk-6u24-linux-i586.bin, file size 84927175k, as shown in Figure 4.5.

Figure 4-5

We can then view the uploaded file at the original terminal, which is in the/usr/local/hadoop/tmp/dfs/data directory of the Linux file system, as shown in Figure 4.6

Figure 4.6

The file that starts with "Blk_" in the figure above is the block that stores the data. The naming here is regular, in addition to the block file, there is a suffix is "meta" file, this is the source data file blocks, storage of some metadata information. Therefore, there are only 2 block files in the above illustration.

Note: We upload a complete file from Linux disk to HDFs, this file is visible in Linux, but after uploading to HDFs, there will not be a corresponding file exists, but is divided into a lot of blocks exist. And since our Hadoop installation is a pseudo distributed installation, only one node, Datanode and Namenode are on this node, so the uploaded block blocks are ultimately in the Linux system. V. The basic structure of HDFs Secondarynode

A solution to ha. But hot standby is not supported. Configuration. Because the more data operations edits file expansion, but can not let him unlimited expansion, so the log process to be converted out into the fsimage. Because Namenode to accept the user's request for action, must be able to quickly respond to user requests, in order to ensure that Namenode fast response to the user, so the work to the Secondarynode, so he also back up part of the fsimage part of the content.

Execution process: Download metadata information (fsimage,edits) from the Namenode, then merge the two, generate a new fsimage, save it locally, and push it to Namenode, Resets the Namenode edits at the same time. The default is installed on the Namenode node, but this ... It's not safe!

The merge principle is shown in Figure 5.1.

Figure 5.1

Original link: http://www.cnblogs.com/sunddenly/p/3977896.html