HDFs System Architecture Detailed

Hadoop is a software platform for developing and running large scale data, and is an open source software framework in the Java language, which realizes the distributed computing of massive data in a large number of computer clusters. Users can develop distributed programs without knowing the underlying details of the distribution. Take full advantage of the power of cluster high speed operation and storage.

The most central design of the Hadoop framework is: HDFs and MapReduce. HDFS provides storage of massive amounts of data, MapReduce provides calculations of data. HDFs plays a very basic role in Hadoop, providing a storage service of massive data to the upper application in the form of file system.

As a distributed file system, HDFs has the characteristics of high fault tolerance. It can be deployed on inexpensive general-purpose hardware, providing high throughput (throughput) data access, especially for applications that need to deal with massive datasets. It does not comply with POSIX requirements, does not support LS,CP UNIX commands such as standard, and does not support file read and write methods such as fopen and Fread. It features a new design that provides a unique set of APIs based on the Hadoop abstract file system that supports streaming access to file system data.

1. Characteristics and limitations of HDFS

HDFs has the following characteristics:

Support for oversized files. In theory, a single file on a hdfs can be stored in all the storage space of the entire cluster.

Fast recovery of hardware failures. Because the cluster contains hundreds of nodes, hardware failure is a very common problem. Fault detection and automatic recovery is an important design goal at the beginning of HDFs design.

Streaming data access. HDFS processing of data scale is relatively large, the application needs to access a large number of data. At the same time, these applications are generally batch processing, rather than user interactive processing. HDFS enables applications to access datasets in stream mode, focusing on data throughput rather than the speed of data access.

A simplified consistency model. Once a HDFs file is created, it can only append and cannot modify existing data. Such a simple consistency model facilitates the provision of high throughput data access.

Due to some of the above design features, HDFs is not suitable for the following applications:

Low latency data access. In the application of user interaction, the application needs to be answered in the time of MS or several s. Because HDFs is designed for high throughput rates, it also sacrifices a quick response. For applications with low latency, hbase or Cassandra can be considered.

A large number of small files. Standard HDFs block size is 64M, storing small files does not waste actual storage space, but will undoubtedly increase the metadata on the Namenode, a large number of small files will affect the performance of the entire cluster. Before we know, btrfs for small files to do the optimization-inline file, for small files have a good space optimization and access time optimization.

Multiple user writes, modifying files. HDFs files can have only one writer, and write operations can only be appended at the end of the file. It does not support multiple writers, nor does it support changes to any location of the file after the file is written.

But in the Big data field, the analysis is the existing data, which will not be modified once it is produced, so the HDFs features and design limitations are easy to understand. HDFs provides a very important and very basic file storage function for data analysis in large data fields.

2. Architecture

2.1 Overview

HDFs uses a master-slave (master/slave) architecture where Namenode (NN), Datanode (DN), and client are the 3 important roles in HDFs. HDFs is also evolving in the community's efforts, including supporting file additions, Federation,ha Introduction, and more. The history of evolution can be clicked here. For HA and Federation, please refer to the blog post "HDFS ha: Historical evolution of a highly reliable distributed storage System solution". This article only deals with secondary namenode (SNN).

In a HDFs, there is a nn, a snn and numerous DN, which may have thousands of DN in a large cluster. And the client, in general sense than the number of data nodes more. The relationship between them is the following diagram:

NN managed the HDFs two most important relationships:

The directory file tree structure and the corresponding relationship between the file and the data block: It is persisted to the physical storage, the filename is called Fsimage.

The DN corresponds to the data block, that is, the DN in which the data block is stored: When the DN is started, the data blocks it maintains are reported over NN. This is dynamically established and will not be persisted. Therefore, it may take a long time for the cluster to start.

The DN preserves the data block. and perform the commands of NN, such as copy, copy, delete and so on.

A client is a common operation that uses HDFS topics, including writing files, reading files, and so on.

2.2 Data blocks

The data blocks are mentioned more than once, so what is a block of data? For ease of management, the device often organizes storage space into a storage unit with a certain structure. For example, the hard disk, the file as a block in the form of storage in the hard disk, the size of the block represents the system read and write operation of the smallest unit; In Linux's Ext3 file system, the block size defaults to 4K. The file system uses a hard disk by an integer multiple of the block size. The data block on the hard disk manages the internal details of the file system implementation and is transparent to users who invoke the system interface to read and write files.

The HDFs block is similar to the principle. But its block has a larger data unit, the default block size is 64M. HDFs blocks the file as a separate storage unit and is stored in the file system of the DN as a regular file on Linux. A block of data is the basic unit of HDFS file storage. Note here that HDFS does not directly manage the hard disk, it is through the DN of the file system to store data blocks, through the management of data blocks, to complete the file Management services.

HDFs is a distributed system designed for large files, and there are many benefits to using blocks of data:

The file size is not limited by the storage space of a single DN, and the file can be saved to any DN.

simplifies the storage subsystem. Simplicity is the design pursuit of all systems. Especially in a wide range of fault distribution systems. Separating management blocks from management files simplifies storage management and eliminates the complexity of distributed management file metadata.

Convenient fault-tolerant, facilitate data replication. In HDFs, data blocks are replicated on different nodes in order to handle damaged blocks as well as hard drives and machine failures. If a copy of a block of data is lost or corrupted, the system reads the copy elsewhere, and the data block is copied to a normally running node, guaranteeing the minimum number of copies of the block. Of course, this process is transparent to the client, and it enables location transparency and fault transparency in distributed systems.

So why is the data block so big? In fact, the large block of data and file system principle is the same. Using large chunks of data in a normal file system, reduces the overhead required to manage blocks of data, such as reducing the length of the chain of information stored in the Hard disk Address table in I-node, and reducing the addressing overhead when writing to a file, that is, the number of times the hard disk locates a block of data. The adoption of a large database in HDFs can reduce the overhead of managing files and data block relationships on NN, and reduces the overhead of establishing a network connection when files are read and written.

This column more highlights: http://www.bianceng.cnhttp://www.bianceng.cn/webkf/tools/