Hadoop distributed system 3

Introduction

HDFS, The hadoop distributed file system, is a distributed system designed to store large amounts of data (usually TB or Pb ), it also provides high-throughput access to data. Files are stored in multiple machines to ensure the system's anti-Failure Performance and the efficiency of parallel applications. This article mainly introduces the design intent and structure of HDFS and tells readers how to use it.

Objectives of this article:

• Understand the design intent of HDFS and some basic concepts of distributed systems.
• Learn how to set up and use HDFS from the command line.
• Learn how to use HDFS in applications.

Basic Principles of Distributed Systems

A Distributed System is designed to store and manage large amounts of data and provide external access to the data (through the network ). Many distributed systems have already solved this problem in different ways.

 

 

NFS,The Network File System is currently the most common distributed system. It is also one of the oldest distributed systems in use. Its design is very understandable, but it also has many limitations. NFS provides remote access to a logical volume stored on a machine. An NFS can make part of its local file system visible to other clients. Then, the client adds the remote system to their own Linux File System and uses it just like the remote system on the local hard disk.

 

 

A major advantage of this model is its transparency. Clients do not need to know whether they are using remote systems. Methods In the standard library, such as open (), close (), fread (), will help us use files on NFS.

 

 

However, as a distributed system, it is not very powerful. All files in NFS are stored only on one machine. This means that the storage capacity cannot exceed the capacity of that machine, and NFS does not provide reliability assurance (such as data backup ). Finally, because all the data is stored on one machine, all clients have to run to this machine to retrieve the data. When many clients run at the same time, the server may be overloaded. In addition, each time the client processes data, it must go to the server to retrieve the data.

 

The HDFS design can solve some other distributed systems, such as NFS, which are powerless. Especially in the following aspects:

• HDFS is designed to store massive amounts of data (usually TB or Pb), which requires that data be stored on multiple machines in a distributed manner. In addition, the file size supported by NFS is much larger than that supported by NFS.
• HDFS can store data reliably, that is, faults of individual machines in the cluster will not affect data usage.
• HDFS provides fast and scalable access to data. It can solve the problem of simultaneous access by a large number of clients by simply adding a machine to the cluster.
• HDFS works well with the hadoop mapreduce programming model to make data computing and reading operations run on the same machine as much as possible.

 

The scalability and high performance of HDFS also determine its strict requirements for applications different from those of other distributed systems. In order to achieve the purpose of the designer, the designer made some additional restrictions on the design and compromise. Mainly include:

 

• The way HDFS applications read files is assumed to be stream reading. HDFS optimizes the performance of stream reading. Of course, this means that it takes a long time to perform random read operations on files.
• HDFS data is designed to allow only one write and multiple read operations. When the write operation is disabled, it is not supported to update some content in the file. (The latest hadoop0.19 will support adding data at the end of the file.
• Because the stored file is too large and Its stream reading method is used, the system does not provide the cache function.
• It is assumed that permanent crashes and intermittent failures of individual machines always occur frequently. The cluster should be able to withstand faults of multiple machines, even if they occur together. The cluster performance will be reduced according to the proportion of the number of lost machines. The whole system will not slow down and data will not be lost. Data backup can be used.

HDFS is designed based on Google's distributed system gfs. This is a paper published by Google.

 

HDFS is a block structure file system. Each file is divided into fixed size file blocks. These blocks are stored on one or more machines according to the data storage policy. The machines in the cluster are called datanode. A file can be divided into several file blocks. These file blocks are not necessarily stored on the same machine, and the block on which it exists is random. Therefore, it may take several machines to work together to access a file. However, it is obvious that files can be stored in a larger space than a single hard disk.

 

If the use of a file requires the cooperation of multiple machines, the failure of one machine will cause the file to become unavailable. HDFS backs up each block of a file to solve this problem (usually three backups exist ).

Figure 2.1: datanodes stores the file blocks. The number of backups is 2. The namenode node maps the file name to the ID of the file block.

 

Most block-structured file systems use 4 K or 8 K block sizes. In contrast, the HDFS block size is 64 MB by default-a large quantity. This design reduces the number of file blocks held by HDFS (when the file block capacity is large, the number of blocks will be reduced accordingly ). This is more conducive to stream reading. Obviously, HDFS prefers extremely large files and stream reading them. Unlike file systems such as NTFS or ext, HDFS usually stores many small files. HDFS prefers to store a moderate number of extra large files, which are hundreds of MB and hundreds of GB. After all, a M file is only two file blocks. In our normal computer, files are usually randomly accessed, and applications may read different parts of a file, which are not stored continuously on the hard disk. In contrast, HDFS expects the program to read the entire file at a time. This method is suitable for the mapreduce programming style. That is to say, using HDFS like a common distributed system is not a wise choice.

 

HDFS divides files into blocks and stores them on several machines. These files cannot be considered part of a normal file system. Use the LS command on a hadoop running machine. The returned result is the content of the Linux system, and does not include any files stored in the HDFS system. HDFS is an application built on a local file system. HDFS files (more specifically, those file blocks that constitute files) are stored in a specific directory of the datanode node, but these file blocks only have IDs. Therefore, you cannot use Linux File System Tools (such as LS, CP, and MV) to operate these files. Don't worry. HDFS comes with the file management function, which is as simple as (LS, CP, MV) and other commands.

 

Data reliability is very important. HDFS data is designed to only allow one write and multiple read operations. When a large number of clients need to modify the file directory at the same time, synchronization becomes very important. Therefore, maintenance of the file directory is done by a single machine, which is called namenode. Namenode stores the file directory and name of the file system. Because each file requires few records (such as file names, permissions, and locations of file blocks), all data can be stored on one machine, and provides quick access.

 

Assume that the client needs to access file. The client obtains the location list of the file blocks that constitute file a from namenode. This list shows the machine on which the file block is stored. Then the client directly reads the file data from datanode. Namenode is not involved in file transmission. We need to ensure that namenode consumes as little as possible.

 

Of course, we also prevent the crash of the namenode machine. We use redundancy to ensure that data in the file system is not lost, even when namenode suddenly crashes. Obviously, the crash of namenode is much more serious than that of any datanode in the cluster. When a datanode fails, the entire cluster can continue to run, but when the namenode fails, the cluster cannot run. At this time, we have to manually take measures to fix the fault. Of course, the workload of namenode is quite small, and the probability of failure is much lower than that of other machines.

 

The following article describes the design and implementation of HDFS in more detail. Document