Introduction to new features of Hadoop 2.4.0

In http://www.aliyun.com/zixun/aggregation/33721.html ">2014 year April 7, Apache released the Hadoop 2.4.0. This version has improved somewhat compared to Hadoop 2.3.0, highlighting changes that can be summed up in the following points (official documentation):

1 support for HDFS access control list (Acl,access controls Lists)

This feature solves the problem of permission access to file permissions under certain circumstances. The mechanism is based on the characteristics of Linux file access, if you are familiar with the Linux file access mechanism, you do not have to understand the characteristics of HDFs file access.

With the ACL characteristics, the HDFs file system has a benign extension of the characteristics. HDFS-4685 bugs that have been resolved in this release.

2 Local support HDFs rolling online upgrade

Problem Resolution (HDFS-5535): "In order to roll a new HDFS release through a SCM cluster quickly and safely, a few enhancements are needed in H DFS. An initial high level design document would be checkmark to this jira, and Sub-jiras would itemize the individual tasks.

3 provides protocol caching for HDFs Fsimage (protocol-buffers)

This feature makes the HDFS upgrade service smoother. Fix the problem (HDFS-5698): "Branch for using PROTOBUF serialization for Fsimage"

4 support for HDFs HTTPS access

5 support yarn ResourceManager fault tolerance

Supports only ResourceManager failed to suspend the reboot, you can restore the previously running applications (users do not need to resubmit), but running and not running tasks need to rerun. In addition, this version does not support ResourceManager primary-standby switching, or even the provisioning ResourceManager. Therefore, the feature is not completely completed. If you want to use it, be aware of its implementation progress.

6 enhances the ability to yarn on new applications

Creator Timeline server runs in the calculation framework on yarn, only MapReduce is equipped with job History server, which allows users to query for information about the completed jobs that have been run. With the increase of the computing framework on the yarn, it is necessary to add a generic job History server, and then developed generic History server, later renamed creator Timeline Server, See Related documentation: Creator Timeline Server. Note: Creator Timeline server can assume that yarn's shared storage module, which is provided to the application for shared information, can store information such as metric in the module, not just historical job-run information. At present, the shared storage module is a stand-alone version of the LEVELDB, users can be extended to hbase as needed.

7 support yarn on Capacityscheduler SLAs

Capacity Scheduler Support Resource preemption This feature has been available for a long time, but has not been adequately tested. This version is fully tested and validated. Here's a quick explanation of the design motivation for capacity Scheduler resource preemption: In Capacity Scheduler, the remaining resources are shared between queues, which can be shared to other queues when resources for a queue are left, but when a new job is submitted for that queue, Other queues must return (release) resources within a certain amount of time, and if they are not returned, the scheduler will preempt them.