Next-generation Hadoop strategy-data platform

Guide: At last week's strata conference, the participants revealed information about the next generation of Hadoop. This includes support for 6000 nodes, HDFs with high availability, and the characteristics of the next generation of MapReduce. And the bold idea of the future of Hadoop as an alternative to the operating system also fills us with endless reverie.

At last week's strata Congress revealed some news about the next generation of Hadoop. As you can hear, the latest version of Hadoop will change much more than it did at any time. The mysterious hype about the size and speed of new features will fundamentally change the way Hadoop is used.

At the strata meeting http://www.aliyun.com/zixun/aggregation/14112.html ">hortonworks founder and Apache Hadoop Project VP Arun Murthy revealed the features of the new version of Hadoop. The next generation of Hadoop will support 6000 nodes with a high-availability HDFs (Hadoop distributed File System).

The next-generation MapReduce framework is named yarn or MRv2 (MapReduce version 2). The new framework is no longer a traditional mapreduce framework, not even mapreduce, and yarn can be viewed as a generic runtime framework in which users can write their own computing frameworks to run in that run environment. The improved MapReduce will separate the management of blocks and namespaces in HDFs and improve the scalability of the data. Most importantly, the improved MapReduce will support mapreduce applications, which will enable more types of applications to be able to work in MapReduce.

Under the new yarn framework, Jobtracker will have two important functions-resource management and task scheduling/monitoring-and dividing them into separate processes. Both have global RM (ResourceManager) and am (Applicationmaster).

ResourceManager and NodeManager together constitute the entire data computing framework. At the same time ResourceManager is the final decision maker that allocates resources to each application in the system. Applicationmaster is actually a concrete framework library, Applicationmaster is primarily responsible for negotiating with the scheduler to get the appropriate containers and tracking the status of these containers and monitoring their progress.

Hadoop provides a reason for businesses to expand their infrastructure, while Hadoop is also a great potential for building data-driven businesses and SaaS applications. In the future, if the application framework is robust enough, a large number of developers will be developing applications on the Hadoop platform. But Hadoop has no intention of replacing the underlying operating system, and instead Hadoop will be a layer between the operating system and the application.

Another bold assumption is that the future Hadoop application framework and file system can be extended to bare metal or other peripherals and effectively become a fully data-oriented operating system. Although there is still a long way to go. (Li/compiling)

(Responsible editor: The good of the Legacy)