Architecture of HDFs

I. HDFS INTRODUCTION

1.1 Background

With the increasing amount of data, in an operating system jurisdiction of the scope of storage, then allocated to more operating system management of the disk, but not easy to manage and maintain, there is an urgent need for a system to manage the files on multiple machines, this is the Distributed file Management system.

The academic point is that a distributed file system is a system that allows files to be shared across multiple hosts over a network, allowing multiple users on multiple machines to share files and storage space. There are many distributed file management systems, and HDFS is just one of them. Applies to the case of one write, multiple queries, does not support concurrent write situations, small files are inappropriate. Because small files also occupy a block, the more small files (1000 1k files) The more blocks, the greater the namenode pressure.

What is 1.2 HDFs?

The files we upload through the Hadoop shell are stored in the Datanode block, which is invisible to the Linux shell, and only blocks are visible to the block. You can describe HDFs in one sentence: The large files of the client are stored in the data blocks of many nodes. Here, there are three keywords : File, node, data block. HDFs is around the three key words designed, we are learning when it is important to seize the three key words to learn.

Two. Basic structure of HDFS

2.1 NameNode

(1) Overview

The function of Namenode is to manage the file directory structure, accept the user's operation request and be the management data node. The name node maintains two sets of data:

① The relationship between the file directory and the data block . is static, stored on disk , and maintained through fsimage and edits files.

② The relationship between the data block and the node . It is not persisted to disk and is automatically created whenever the cluster is started, so it is usually placed in memory .

So he is the management node for the entire file system. It maintains the file directory tree of the entire file system, the meta-information of the file/directory, and the list of data blocks corresponding to each file, receiving the user's action request.

Documents include:

① fsimage (File system image): Metadata image file. Stores Namenode memory metadata information for a certain period of time.

② edits: operation log file.

③ fstime: Time to save last checkpoint

These files are stored in the Linux file system

(2) Features

is a file system that allows file networks to be shared across multiple hosts, allowing multiple users on multiple machines to share files and storage space.

① permeability. Let's actually access the file through the network action, by the program and the user, it is like accessing the local disk generally.

② fault tolerance. Even if some nodes are offline in the system, the system can continue to operate without any data loss as a whole.

③ is suitable for one-time write, multiple queries, does not support concurrent write situations, small files inappropriate

(3) directory structure

a) since Namenode maintains so much information, where is this information stored?

In the Hadoop source code, there is a file called hdfs-default.xml,3.1.

Figure 3.1

b) Open this file

In lines 149th and 158th, there are two configuration information, one is Dfs.name.dir and the other is Dfs.name.edits.dir. These two files represent the location of the Namenode core files Fsimage and edits, as shown in 3.2.

Figure 3.2

The value of the corresponding configuration has ${}, which is the representation of the variable, er expression, when the program reads the file, the value of the variable will be read out. So, the value of the variable Hadoop.tmp.dir in line 150th (that is, the Hadoop temporary storage path), 3.3.

Figure 3.3

But in our previous chapter, in the configuration file Core-site.xml, the value configured is/usr/local/hadoop/tmp.

c) We can access the Linux file system

Execute the command Cd/usr/local/hadoop/conf,more core-site.xml see the content shown in 3.3.

Figure 3.4

As you can see, these two files are stored in the/usr/local/hadoop/tmp/dfs/name directory of the Linux file system.

d) We enter this directory

View the contents of this directory, as shown in 3.5.

Figure 3.5

It is known that the core files of Namenode fsimage and edits are stored in the current directory, while the name directory has a file In_ Use.lock while viewing its content, the content is empty, that is, only one namenode process can access the directory, readers can try it yourself, when there is no Hadoop, there is no file In_use.lock in the directory, This file will not be generated until Hadoop is started.

e) document Fsimage pieces

This file is very important, if lost, namenode can not be used, then how to prevent the loss of the file and cause undesirable consequences. I can take a look at the Hdfs-default.xml section of code 3.6 below.

Figure 3.6

This variable, as described in the description, determines where the DFS NameNode nametable (fsimage) should be stored on the local file system. If this is a comma-delimited list of directories, then NameTable will be replicated to all directories for redundancy (backup to ensure data security). such as ${hadoop.tmp.dir}/dfs/name,~/name2,~/name3,~/name4. Then the fsimage will be copied to the ~/name1,~/name2,~/name3,~/name4 directory, respectively. So these directories are generally on different machines, different disks, different folders, in short, the more dispersed the better, so as to ensure the security of the data. Some people will ask how to implement on multiple machines? In fact, there is NFS file sharing system in Linux, not detailed here.

f) look at the description of edits

Take a look at a section of code 3.7 in Hdfs-default.xml

Figure 3.7

This variable, as described in the description, determines the location of the Dfsnamenode storage transaction file (edits) on the local file system. If this is a comma-delimited list of directories, then the transaction file will be duplicated in all directories to be redundant. The default value is Dfs.name.dir. (Edit Save Transaction Procedure)

2.2 DataNode

(1) Overview

The role of Datanode is to actually store data in HDFs.

(2) block

① If a file is very large, such as 100GB, how is it stored in Datanode? Datanode data is stored in block-based reading and writing. Block is the basic unit of HDFs read and write data.

② assumes that the file size is 100GB, starting at byte position 0, each 64MB byte is divided into a block, and so on, can be divided into a lot of block. Each block is 64MB in size.

a) let's take a look at the Org.apache.hadoop.hdfs.protocol.Block class, which has the following properties, shown in 4.1.

Figure 4.1

It is known that none of the properties in a class can store data. So block is essentially a logical concept, meaning that the block does not actually store the data, it just divides the files.

b) Why must it be divided into 64MB size?

Since this is set in the default profile, we view the Core-default.xml file as shown in 4.2.

Figure 4.2

The parameter in the Ds.block.name refers to the size of the block, the value is 67 108 864 bytes, can be converted to 64MB. If we don't want to use a 64MB size, we can override that value in Core-site.xml. Note that units are bytes.

(3) Copy

a) A copy is a backup for the purpose of security. Because the cluster environment is unreliable, the replica mechanism is used to ensure the security of the data.

b) The disadvantage of a replica is that it consumes a lot of storage space. The more replicas, the more space they occupy. The cost of storage space is worthwhile compared to the risk of data loss.

c) So, how many copies of a file are appropriate? We can look at the Hdfs-default.xml file as shown in 4.3.

Figure 4.3

As you can see from Figure 4.3, the default number of replicas is 3. means that each chunk of data in HDFs has 3 copies. Of course, each one will definitely try to distribute it in a different Datanode server. Imagine: If the backup of the 3 data are on the same server, then the server is down, is not all the data are lost AH?

(4) directory structure

a) Datanode is divided by block.

So where exactly is the partitioned file stored? We look at the file as shown in core-default.xml,4.4.

Figure 4.4

The value of the parameter Dfs.data.dir is the location of the block stored in the Linux file system. The value of the variable Hadoop.tmp.dir is described earlier and is/usr/local/hadoop/tmp, so the full path to Dfs.data.dir is/usr/local/hadoop/tmp/dfs/data. Viewed from the Linux command, as shown in result 4.5.

b) upload a file

We first click Pietty to open another Linux terminal, upload a file jdk-6u24-linux-i586.bin, the file size is shown in 84927175k,4.5.

Figure 4-5

Then we can look at the original terminal, the upload file, is in the Linux file system/usr/local/hadoop/tmp/dfs/data directory, 4.6 shows

Figure 4.6

The file that begins with "Blk_" is the block where the data is stored. The name here is regular, in addition to the block file, there is a suffix "meta" file, which is the block's source data file, storing some meta-data information. Therefore, there are only 2 block files.

Note: We upload a complete file from the Linux disk into HDFs, this file can be seen in Linux, but after uploading to HDFs, there will not be a corresponding file exists, but is divided into a lot of blocks exist. And since our Hadoop installation is a pseudo-distributed installation, with only one node, Datanode and Namenode on this node, the block blocks that are uploaded are ultimately in the Linux system.

2.3 Secondarynamenode

A solution for ha. But it does not support hot standby. Configuration. As more data operations edits file expansion, but can not allow him to expand indefinitely, so to convert the log process to put into the fsimage. Because Namenode to accept the user's operation request, must be able to quickly respond to the user request, in order to ensure the Namenode quick response to the user, so the work to Secondarynode, so he also backed up part of the fsimage portion of the content.

Execution process: Download metadata information (fsimage,edits) from Namenode, then merge the two, generate a new fsimage, save it locally, and push it to Namenode, Resets the edits of Namenode at the same time. The default is installed on the Namenode node, but this ... Not Safe!

Merge principle 5.1 is shown.

When do I trigger a merger?

Divided into time trigger and size trigger:

<property>

<name>fs.checkpoint.period</name>

<value>3600</value>

<description>the number of seconds between and periodic checkpoints.

</description>

</property>

<property>

<name>fs.checkpoint.size</name>

<value>67108864</value>

<description>the size of the current edit log (in bytes) that triggers

A periodic checkpoint even if the fs.checkpoint.period hasn ' t expired.

</description>

</property>

What happens when I perform the format of HDFs?

Namenode Create your own directory structure

View Dfs/name/current under Version

Namespaceid?

View/dfs/data/current VERSION of other nodes of the cluster

Exactly the same, when multiple formatting will produce different id,namenode will change, Datanode will not change

So you can't store the data. Please change Namespaceid If there is a mistake.

Architecture of HDFs