Hadoop is more suitable for enterprise application when it enters 2.0 times

"Now is the best time for companies to apply Hadoop. "Jeff Markham, Hortonworks's chief technology officer, said in a speech at the 2013 China Hadoop Technology Summit, which was held at the end of November. At this summit, Hadoop entered the 2.0 era as the focus of people's talk. Jeff Markham says Hadoop 2.0 has a stronger, broader range of new features that meet the needs of enterprise users, making up for the lack of Hadoop 1.0 and more in line with the needs of business users.

Hadoop makeover

When Jeff Markham introduced the new features of Hadoop 2.0, the reporter heard a whisper behind him: "You see, the Hadoop 2.0 frame has a few more strange features." "Yes, the most important of these functional modules is yarn." Yarn is actually a resource manager, which, in a way, subverts the data processing core of Hadoop MapReduce, allowing users to run Hadoop in a new way that is completely different from batch processing. As we all know, Hadoop was designed to search and index Web pages, and MapReduce, who handles data, is good at handling and analyzing unstructured or semi-structured data, such as log files, but not for all types of data. As data volumes grow and data complexity increases, people want to be able to handle multiple types of applications in a single cluster. This is also the background of the birth of Hadoop 2.0.

Some people think that yarn is essentially Hadoop's new operating system, which breaks through the mapreduce performance bottleneck. The combination of Hadoop and yarn is more suitable for enterprise large data application. Yarn's design idea is to separate the resource management from the job scheduling/monitoring function, and its architecture is achieved through a global ResourceManager with several specific application-oriented applicationmaster combinations, Where ResourceManager is responsible for assigning resources to individual applications, and Applicationmaster is responsible for running and monitoring tasks. "By joining the yarn management, Hadoop can better meet the needs of enterprise-level users for large data platforms," says Jeff Markham. Our company from the security, management, configuration and many other levels have been ready for Hadoop 2.0 to enter the enterprise. ”

Hadoop 2.0 is no longer an idea, but a real solution. Star-Ring Information Technology (Shanghai) Co., Ltd., a large data company based in China, announced at the summit that it was officially launching the TRANSWARP data Hub of the spark and Hadoop 2.0. "A common idea for enterprise users is to deal more efficiently with larger amounts of data while reducing delays." "In the past, people used different processing techniques, such as memory technology, indexing technology and some performance optimization techniques, for different levels of data," said Sun Yuanhao, founder and CTO of Star Ring technology. One of the most prominent advantages of the TRANSWARP data hub is that data from gigabytes to petabytes can be processed on a single platform. ”

It is because the Transwarp Data hub has the ability to use a wide range of applications, including off-line analysis, statistics and mining, online storage and online memory based high-speed analysis. The TRANSWARP Data hub integrates integrated/etl, large data storage and online service systems, an efficient memory-based computing engine, high-performance SQL, statistical analysis, and machine learning to achieve a breakthrough in performance. In Sun Yuanhao's words, the transwarp Data hub has a "lightning" speed that is 10~100 times faster than open source Hadoop 2.0. In addition, the Transwarp Data Hub has strong analytical capabilities and is fully compatible with the Hadoop ecosystem.

To Transwarp data hub as the core, star-ring technology and many large data manufacturers have cooperated, including Revolution R, Informatica, tableau and so on, these manufacturers of data processing and analysis tools to integrate, constitute a complete large data platform.

Lower application threshold

Because of the complexity of Hadoop itself and the lack of relevant large data professionals in the enterprise, it is not easy for Hadoop to be quickly popularized among enterprise users. As a result, many IT vendors are throwing "olive branches" at Hadoop, some offering a hardware solution based on Hadoop, and others launching the commercial release of the Hadoop software, with the goal of reducing the application threshold of Hadoop.

At this summit, many well-known it vendors, including Intel, VMware, Huawei, and many other telecom operators, internet companies have their own views, for Hadoop in China's promotional station feet. He Jingxiang, general manager of Intel Asia Pacific Research and Development Co., said that, in addition to releasing the Hadoop commercial release, Intel provides full support for Hadoop in terms of hardware (including processors, solid-state drives, etc.), security, management, and optimization to make hadoop more responsive to the needs of enterprise users.