Hadoop 2 finally comes out: Big data takes a giant step forward

The

Apache Software Foundation has finally launched the latest Hadoop 2 data analysis platform. Hadoop 2 enhances its computing engine by supporting the yarn data processing and service engine while adding highly available features to the Hadoop File System (HDFS).
Although HDFs has been upgraded in some Hadoop distributions, such as Cloudera, and some companies such as pivotal have been providing yarn support for half a year, Apache's public release of this version will provide greater confidence in the user's handling of the data. Milind Bhandarkar, chief scientist of the pivotal company, said: "The fully released functionality allows users to ensure that these user-oriented APIs and yarn protocols are stable and will not change until the next generation of Hadoop's major version appears, It is also still more reassuring to build applications and use these APIs. The

The

yarn has changed dramatically, changing the way the Hadoop computing component (MapReduce) splits and processes tasks, because yarn cuts MapReduce's tracking components into two different parts: the resource Manager and the application schedule.
This makes it easier for the data-finishing tools to run tasks such as MapReduce or Storm at the same time, as well as services such as HBase. "It makes other mapreduce workloads now more efficient at sharing resources with MapReduce," says Doug Cutting, a co-founder of Hadoop. Now these systems can share resources dynamically, and resources can also be prioritized. "
Cuuting and Bhandarkar acknowledge that this approach is influenced by the Apache project" Mesos "Cluster management system and Google Borg and Omega secret projects.
Bhandarkar says: "I have to say that on the one hand the Borg/omega framework is a slightly lower level framework for resource allocation and resource management." On the other hand, Borg/omega can do better on the scale of the data center than yarn. "What
yarn can bring to Hadoop is the ability to turn Hadoop into a more native platform, run a lot of data-driven applications and services, and help transform the Hadoop system from a data-processing system into a software ecosystem that covers the entire data center operating system," cutting said. "Yarn opens up the distributed processing power of Hadoop to make it more customizable and more scalable than the initial deployment focused on MapReduce," says James Watters, Cloud Foundry, the pivotal company. Another feature added to
Hadoop 2 is HDFs Federation, which allows a hdfs to have multiple namespaces within a cluster. This increases the availability of the system as a whole, leaving different applications to each other and improving file system throughput by eliminating a single named node bottleneck.
Cutting predicts that Hadoop will have a bright future and is amazed at how many years has passed since the first yellow elephant was born in early 21st century.
Now it has grown into a data center operating system that supports a wide range of applications--that I could not have imagined. I am confident that open source will be the best way to release Hadoop technology and promote popularity. The