Intel adds lustre support capabilities to Hadoop

The world's manufacturers have reached a consensus: Hadoop is a very good tool in the mapping of simplification, but the software's further development is subject to a variety of constraints, the most difficult hurdle to cross the Hadoop Distributed File system (referred to as HDFS) highly dependent.


HDFs itself is fine, but when integrated with Hadoop requires users to build a dedicated computer cluster for them.


Although we are not overly resistant to HDFs, most customers who use high-performance computing clusters to handle special transactions tend to be less enthusiastic about it. The reason is that the user needs to devote a lot of computing resources to HDFS itself. Although the mapping simplification feature does bring some convenience to task execution, this part of the resource does not directly affect the operation of Hadoop.


Intel has noted this shortcoming and has added support for lustre in a quietly released version of its Hadoop release version 2.5 last week.


Http://www.aliyun.com/zixun/aggregation/18652.html, general manager of the large data and software services Division of Intel > Corporation Girish Juneja, said The chip giant's high-performance computing customers are raving about the new scheme. And Intel's decision to fully promote open source rules in the release will not affect other customers.


"Many customers do not want to deploy a complete set of independent physical clusters, mainly because they do not know how to run Hadoop in their own file system," Juneja in Ho Chi Minh City, Vietnam, said at the Intel large data and Yunfeng meeting. "High-performance Computing is the most immediate beneficiary of the latest decision." In the field of High-performance computing, many users are using GPFS or lustre, and we are pleased to be able to introduce lustre in our own business. ”


"We construct the HDFS layer in an abstract form, but in essence it still belongs to lustre." ”


"Therefore, we might as well be concerned about the use of research environments such as Los Alamos Laboratories." In existing cluster facilities, the device has more than 90% of the time to run High-performance computing tasks, but for the remaining 10% use time, technicians can run the Hadoop task-the entire process does not involve any data migrations and is fully implemented in the same environment. ”


in view of the fact that such laboratories often have to deal with large amounts of data, such data-keeping programs will certainly be popular.


Chip giants also show concern for HBase's encryption and control list access.


"The biggest challenge for technicians in a nosql environment is how to specify which users have access to which data," Juneja said. "We provide additional functionality to implement control list access", which allows administrators to set data access rights policies in HBase.


In addition, Juneja believes that the introduction of encryption and data anonymity can prompt financial service providers and users who have previously been worried about compliance burdens to consider investing in Hadoop. Juneja points out that the lack of such security functionality in the past means that Hadoop can lead to unacceptable risks.


Intel also sells its own management software to drive access control lists. In Juneja's view, this does not incur customer aversion.

The
chip giant's Hadoop release 3.0 will soon meet with users and should be officially released in September, according to the current situation. Juneja says users can expect Intel to end up with an excellent release that is cohesive to the Hadoop community.