014_HDFS storage architecture, architecture reliability Analysis, copy placement strategy, relationships between components

1.HDFS Storage Architecture

(1) HDFS Architecture--File

1) The file is cut into chunks (default size 64M), in blocks, each block has multiple copies stored on different machines, the number of copies can be specified at the time of file generation (silent
Recognition 3)
2) NameNode is the master node that stores metadata such as file name, file directory structure, file attributes (generation time, number of copies, file permissions), and each
The block list of the file and the Datanode of the block, etc.
3) DataNode stores the file block data in the local file system, as well as the checksum of the block data .
4) You can create, delete, move, or rename files, and you cannot modify the contents of a file after it is created, written, and closed.

(2) HDFS file Permissions

1) Similar to Linux file permissions.
2) R:read; W:write; X:execute, permissions x is ignored for files, and the folder indicates whether to allow access to its contents.
3) If the Linux system user Zhangsan to create a file using the Hadoop command, the file in HDFs is owner Zhangsan.
4) HDFs Permissions Purpose: To prevent good people from doing wrong, not to stop bad people from doing bad things . HDFs believe that if you tell me who you are, I think you are who you are.

(3) HDFS Architecture-Component functionality

(4) HDFS Architecture--namenode

1) Namenode is a central server, a single node (simplified system design and implementation), responsible for managing the file System namespace (namespace) and client access to files.
2) file operation,NameNode is responsible for the operation of the file metadata, Datanode is responsible for processing the file content of the read and write requests , with the file content related to the data flow does not go through the NameNode, will only ask it to contact the Datanode, Otherwise Namenode will become the bottleneck of the system.

3) The copy is stored on which Datanode is controlled by the Namenode, according to the global situation to make block placement decision, read the file Namenode try to let the user read the most recent copy, reduce the block consumption and read latency.
4) Namenode fully manages the replication of the data block, which periodically receives heartbeat and block status reports (Blockreport) from each datanode in the cluster. Receiving a heartbeat signal means that the Datanode node is working properly. The Block status report contains a list of all the data blocks on the Datanode.

(5) HDFS Architecture--datanode

1) a data block is stored on disk in Datanode, including two files , one is the data itself , the other is the length of the metadata including the block, the checksum of the block data, and the timestamp
2) Datanode to Namenode after the start of registration, through the periodic (1 hours) to the Namenode report all the block information.
3) Heartbeat is every 3 seconds, heartbeat returns results with Namenode to the Datanode command such as copying block data to another machine, or deleting a block of data. If a datanode heartbeat is not received for more than 10 minutes, the node is considered unavailable.
4) The operation of the cluster can safely join and exit some machines

2.HDFS Architecture Reliability (1) Measures to guarantee reliability

1) A name node and multiple data nodes
2) data replication (redundancy mechanism)
Location of storage (rack-aware policy)
3) Fault Detection
Data node
Heartbeat Pack (check for downtime)
Block reporting (detection in safe mode)
Data integrity Detection (checksum comparison)
Name node (log file, image file)
4) Space recovery mechanism

(2) Common three error cases and solutions

1) file corruption 2) network or machine failure 3) Namenode hang up

　　

(3) Data Corruption (corruption) processing

1) When Datanode reads the block, it calculates the checksum
2) If the computed checksum is not the same as the block creation value, it indicates that the block is corrupted.
3) The client reads the block on the other DN.
4) Namenode mark the block has been corrupted, and then copy the block to achieve the desired set of file backup number.
5) DataNode validates its checksum for three weeks after its file is created.

3. Copy Placement Policy

　　

Architecture diagram between the client, NN, DN, snn Relationship (1) in the 4.HDFS architecture

(2) HDFS Architecture--client & SNN

(3) HDFS Architecture--nn & SNN

Secondnamenode: Not Namenode hot backup, is auxiliary namenode, share its workload, regularly merge fsimage and dsedits, push to Namenode, in emergency, can assist recovery namenode.
When the cluster starts, the fsimage is loaded into Namenode memory, and the operation is saved in Fsedits.
Namenode memory to keep a copy of the latest, in-memory image files equal to Fsimage+edits, regularly merge fsimage and edits, otherwise, because the edits file too long cause namenode restart slow. Secondnamenode is responsible for periodically merging them.

Description: This blog is basically all organized from the cloud sail Big Data Dream Qi teacher ppt

014_HDFS storage architecture, architecture reliability Analysis, copy placement strategy, relationships between components