004, Hadoop-hdfs Distributed File system detailed

Official API link Address: http://hadoop.apache.org/docs/current/

First, what is HDFs?

HDFS (Hadoop Distributed File System): The universal Distributed File system above Hadoop, with high fault tolerance, high throughput features, and it is also at the heart of Hadoop.

Ii. advantages and disadvantages of Hadoop

Advantages:

1) High reliability. Hadoop's ability to store and process data in bits is worth trusting;

2) High scalability. Hadoop distributes data between available computer sets and completes computational tasks, and these clusters can be easily extended to thousands of nodes.

3) High efficiency. Hadoop can dynamically move data between nodes and ensure the dynamic balance of individual nodes, so processing is very fast.

4) high fault tolerance. Hadoop has the ability to automatically save multiple copies of data and automatically reassign failed tasks.

Disadvantages:

1) Not suitable for low latency data access.

2) cannot efficiently store large numbers of small files.

3) does not support multi-user write and arbitrary modification of files.

Third, the operation of the HDFs command

Hadoop FS + commands below

-ls <path>: View the files inside HDFs

-LSR <path>: Recursively view files in HDFs

-mkdir <path>: Create a folder in HDFs

-put <path>

-get

-text <path>: View the contents of a file inside HDFs

-rm[r] <path>: Delete files inside HDFs

Iv. Namenode of HDFs

It is the management node of the entire file system, which maintains a directory tree of the entire file system, meta-information for files/directories, and a list of data blocks corresponding to each file. Accepts the user's action request.

1) Fsimage: Metadata image file, storing Namenode memory metadata information for a certain period of time.

2) Edits: operation log file.

3) Fstime: Save the last checkpoint time

Note: The above is three files stored in the Linux file system

V. Secondarynamenode in HDFs

1) Download the data (Fsimage,edit) from the Namenode, then merge the two, generate a new fsimage, save it locally, and push it to Namenode, while resetting the edit in Namenode

Vi. Datanode of HDFs

1) File Block: The most basic unit of storage. For the file content, the length of a file is size, then starting from the 0 offset of the file, according to the fixed size, order the number of files, divided each block called a BLOCK.HDFS default block size is 128M.

2) Unlike the normal file system, HDFs, if a file is smaller than the size of a database, does not occupy the entire chunk of storage space.

004, Hadoop-hdfs Distributed File system detailed