The architecture and principle of HDFs

The HDFS (Hadoop Distributed File System) is one of the core components of Hadoop and is the basis for data storage management in distributed computing, and is designed to be suitable for distributed file systems running on common hardware. HDFS architecture has two types of nodes, one is Namenode, also known as "meta-data Node", the other is Datanode, also known as "Data Node", respectively, to perform the specific tasks of master and worker.

HDFs is a (master/slave) architecture, "write once, read multiple times." HDFs design idea: divide and conquer-large files, large batches of files distributed on a large number of independent machines.

I. Advantages and disadvantages of HDFs

Advantages:

1, high fault tolerance. Data holds multiple copies, increases fault tolerance by increasing the number of replicas, and can be automatically recovered by other replicas once a replica is lost.

2, suitable for large-scale data processing. Processing up to GB, TB, or even petabytes of data, processing the number of millions of files or more, processing the size of 10K nodes.

3, streaming file access. Once write multiple reads, once the file can not be modified, only append, ensure data consistency.

4, can be built on the cheap machine. Improve reliability through multi-copy mechanism, provide fault tolerance and recovery mechanism.

Disadvantages (scenarios that do not apply to HDFs):

1, low latency data access. does not store data in milliseconds, but is suitable for scenarios where high throughput rates (a large amount of data is written in a given time).

2, small file storage. Storing large amounts of small files consumes namenode of memory to store files, directories, and block information.

3, concurrent write, Random Read and write. A file does not allow multiple threads to write simultaneously, only supports data append, and does not support random modification of files.

Second, HDFs storage architecture

Data Storage Architecture Diagram:

HDFs uses Master/slave architecture to store data, consisting of the HDFs Client, NameNode, Datanode, and secondary NameNode four parts.

Client: Clients

1, file segmentation. When uploading an HDFs file, the client splits the file by block size and then stores

2. Interact with Namenode to get file location information

3. Interacting with Datanode, reading or writing data

4, the client provides some command management and access to HDFs

Namenode:master (manager)

1. Managing the name space of HDFs

2. Manage data block (blocks) mapping information

3. Configure replica Policy

4. Processing client read and write requests

Datanode:slave (NN release command to perform actual operation)

1. Store the actual data block

2. Execute data block read/write operation

Secondary NameNode: Not a NameNode hot standby, when nn stops service, it does not immediately replace the NN and provide services

1, auxiliary NN, share its workload

2. Periodically merge Fsimage and Fsedits, and push to NN

3, in case of emergency, can assist to restore nn

Third, HDFS data read and write

File Read steps:

1, first call FileSystem's Open method to get a Distributedfilesystem instance.

2, Distributedfilesystem through RPC (remote procedure Call) to obtain the first block of the file locations, the same block returns multiple locations according to the number of repetitions, These locations are sorted according to the Hadoop topology and sorted by the nearest principle.

3. A Fsdatainputstream object is returned after the first two steps, and when the Read method is called, the object finds the closest datanode to the client and connects.

4, the data from the Datanode flow to the client.

5. If the first block data read is completed, the Datanode connection to the first block block is closed, and then the next block chunk is read.

6, if the first batch of blocks read completed, Fsdatainputstream will get the next batch of blocks locations to NN, and then repeat 4, 5 steps until all blocks read complete, this will close all the streams.

File Write steps:

1. The client creates a new file by calling the Create method of Distributedfilesystem.

2. Distributedfilesystem calls Namenode via RPC (remote procedure Call) to create a new file with no blocks associated with it. Before the creation, NN will perform various checks, if the verification pass, NN will record the new file, otherwise throw I/O exception.

3, the first two steps will return a Fsdataoutputstream object, the client began to write data to Fsdataoutputstream,fsdataoutputstream will cut the data into a small packet, and then queued to the data queue.

4, Datastreamer will be processed to accept the data queue, it first inquiry NameNode this new block is most suitable for storage in which several DataNode, such as repeat number is 3, then find 3 most suitable DataNode, put them into a Pipeline Datastreamer packet the output to the first DataNode of the pipeline, the first DataNode outputs the packet to the second DataNode, and so on.

5, Dfsoutputstream also has a queue called Ack queue, is also composed of packet, waiting for Datanode to receive a response, when all datanode in pipeline is said to have received, then AKC The queue will remove the corresponding packet packet.

6. After the client finishes writing the data, call the Close method to close the write stream.

7, Datastreamer the remaining packages are brushed into the pipeline, and then wait for the ACK message, after receiving the last Ack, notify NameNode to mark the file as completed.

Rackaware Rack-aware features:

1, if the client is a Datanode node, the storage block, the rule is: Copy 1 is stored on the same client node; Copy 2 is stored on different rack nodes; Copy 3 copy 2 on another node of the rack; other copies are randomly selected.

2, if the client is not a Datanode node, the storage block, the rule is: Copy 1 randomly select a node; Copy 2 is stored in a different rack node than replica 1, and replica 3 is another node in the same rack as replica 2; other copies are randomly selected.

The architecture and principle of HDFs