Hadoop Architecture introduces the architecture of--hdfs _hadoop

Design objectives:

-(Hardware failure is normal, not accidental) automatic rapid detection to deal with hardware errors

-Streaming Access data (data batch processing)

-Transfer calculation is more cost-effective than moving the data itself (reducing data transfer)

-Simple data consistency model (one write, multiple read file access model)

-Heterogeneous Platform portability

HDFS Architecture

Adopt Master-slaver Mode:

Namenode Central Server (Master): Maintains the file system tree, as well as the file directory within the entire tree, and is responsible for the management of the entire data cluster.

Datanode are distributed on different racks (slaver): Stores and retrieves blocks of data under the dispatch of the client or Namenode, and periodically sends a list of the stored blocks of data to Namenode.

The client and the Namenode obtain the meta data;

Get data interactively with Datanode.

By default, each Datanode holds 3 replicas, of which two are saved on two different nodes of the same rack. Another copy is placed on a node on a different rack.

Basic concepts

Rack: HDFs cluster, consisting of a large number of datanode distributed across multiple racks, communication between different racks through switches, HDFS through a rack-aware strategy, enables Namenode to determine the rack ID to which each datanode belongs, and uses a replica storage strategy. To improve data reliability, availability, and network bandwidth utilization.

Block: HDFs is the most basic storage unit, the default is 64M, the user can set the size.

Metadata: Refers to the property information of files and directories in the HDFs file system. The HDFS implementation uses the backup mechanism of the mirrored file (fsimage) + log file (Editlog). The contents of the file's mirrored file include: modification time, access time, block size, storage location information for the block of data that makes up the file. The contents of a mirrored file include: modification time, access control permissions, and other information. Log file records are: HDFs update operations.

When Namenode is started, the contents of the mirrored file and log file are merged in memory. Updates the metadata in memory to the latest status.

User data: Most of the HDFS storage is user data, stored in the form of data blocks on Datanode.

In HDFs, the TCP protocol is used to communicate between Namenode and Datanode. Datanode sends a heartbeat per 3s to the namenode. After every 10 heartbeats, send a block of data to Namenode to report its own information, through which Namenode can reconstruct the metadata and ensure that there are sufficient copies of each block of data.

HDFs the process of writing data:

HDFs the process of reading data: