Hadoop2.0 new Features-continuous addition of "dry goods"

1. NAME NODE HA

2. NAME NODE Federation

3. HDFS Snapshot (Directory snapshot)

4. HDFS Cache

5. HDFS ACL

6. Heterogeneous Hierarchical storage structure

-------------------------------------------------------------------------------------------------------

　1. NameNode HA:

Namenode is the main standby node, one for Activenamenode, and one for Standbynamenode.

Activenamenode Effect:

1, Master master node, only one;

2, management of the HDFs namespace;

3, the mapping relationship between the data block and the data node, and the mapping relationship between the file and the data block;

4, configuration copy policy;

5, processing client requests;

Standbynamenode Effect:

1, Namenode hot standby;

2. Periodically merge meta-data information, including (Fsimage and Fsedits). and pushed to Activenamenode;

3, when the Activenamenode fault, fast switching is called the new Activenamenode;

2, NameNode Federation:

multiple Namenode work together, each namenode in charge of a different HDFs directory, sharing all Datanode data nodes.

3. HDFS Snapshot:

mainly to prevent users from mistakenly deleting data, and the role of data backup

The object of the snapshot is the HDFs directory, provided that the directory is allowed to set the snapshot (snapshotable)

Enable snapshot: Bin/hdfs dfsadmin allowsnapshot <path>

Snapshot creation: Bin/hdfs dfs-createsnapshot <path> [Snapshotname]

Delete Snapshot: Bin/hdfs dfs-deletesnapshot <path> [Snapshotname]

Snapshot features: read-only, non-modifiable

4. HDFS Cache:

Purpose: To allow long-term caching of data accessed by high frequency.

Features: 1, only directory or file cache is supported, block level is not supported

2. Automatic caching is not supported

3. Do not recursively cache all files in directory subdirectory

4. Cache time can be set

5, independent management of memory, has not been integrated with yarn resource management, there may be a conflict of resources allocation ·

5. HDFS ACL:

Objective: To improve the security flexibility of the HDFS directory Rights Management, the original permissions manage the directory rights management of users, user groups, and other groups similar to the Linux operating system pattern.

Configuration: Configured on the Namenode node, dfs.namenode.acls.enabled:true

Example:

HDFs dfs-setfacl-m USER:TOM:RW-/lab/user

HDFs dfs-setfacl-m USER:LUCY:RW-/lab/user

HDFs dfs-setfacl-m Group:team2:r-/lab/user

HDFs dfs-setfacl-m Group:team3:r-/lab/user

6. HDFS Heterogeneous hierarchical storage structure

　　　　　 Different storage media categories, such as disks, SSDs, and RAM, can be configured for different data catalogs to meet the data storage requirements for different computing frameworks, different computing tasks, and different performance requirements for Hadoop clusters.

Configuration:

<property>

<name>dfs.datanode.data.dir</name>

<value>[disk]/dir1,[disk]/dir2,[ssd]/dir3,[ssd]/dir4</value>

</property>

　　　　　HDFS provides APIs to control catalog files, what media to write to, and to limit the available share per user for each medium.

Hadoop2.0 new Features-continuous addition of "dry goods"