IDF2013: Intel's Big Data beyond Hadoop

"IT168" April 10, 2013 news, the 2013 Intel Information Technology Summit (IDF 2013) at the Beijing National Convention Center, the current IDF theme for the "future, with the core" experience, announcing that Intel is more user experience as the core, Based on Intel architecture, we will continue to expand and deepen industrial cooperation, comprehensively promote computing technology innovation, chip manufacturing innovation, application experience innovation, terminal form innovation and cloud intelligent innovation, to open a new era of personalized experience with powerful computing power. Thousands of software and hardware developers, technical managers and media and analysts from all over China and around the world will come together to experience cutting-edge technological achievements and explore future trends.

Intel Developer Forum (intel® Information Technology Summit, IDF) is a technical lecture sponsored by Intel Corporation, held in 7 regions in the United States and China, and held two times a year in spring and autumn. IDF mainly by keynote speeches, technical lectures and technical presentations, the keynote speakers are Intel's top personalities, the presentations are fairly forward-looking, and as a leading company in the processor, network processor, and so on, the IDF is the best place to get the industry to know about Intel's latest trends.

▲IDF 2013 Field Report topics

Hadoop brings mapreduce parallel computing into mainstream applications. However, with the expansion of large data requirements and usage patterns, Hadoop has exposed many limitations. This afternoon, the "Big Data Beyond Hadoop: Future research Direction" course was brought by Dr 勱, Director of engineering and chief engineer Jason Dai, Intel Software and Solutions division, and the research program at Intel University research Collaboration Office. Describes the collaborative research between Intel and university partners to go beyond these constraints, and highlights efforts to apply some of the results to the production environment.

▲ Intel Software and Solutions division engineering Director and chief engineer Jason Dai and director of research programs at Intel University research Collaboration Office Dr. Rui 勱

The industry's understanding of big data is different, in the view of Intel's two experts, large data has a large number, fast, existing systems and algorithms difficult to deal with the characteristics, in particular, the large number refers to the TB level of PB-level, the need for intelligent (rather than strong) large-scale parallel processing; Speed is the ubiquitous sensor brings new mass of data, and ingestion difficulties, difficult to handle refers to the need for complex analysis, and integration of a variety of data types. Jason Dai, the engineering director and Chief engineer of Intel Software and Solutions division, says data should be resources, not loads, and existing data processing tools are not perfect.

▲ Hadoop under Large data ecosystems

Referring to the Berkeley data Analysis system, Dr. Rui 勱 said that Berkeley data analysis system includes three parts: Mesos, resource management platform; Scads, not dependent on the size of the storage system; The Piql,spark processing frame is composed of three parts. The spark is a memory cluster computing framework for the application of the reusable work dataset, whose main idea is: RDD "recoverable, distributed DataSet", which can be rebuilt automatically after failure. Spark a fault-tolerant mechanism based on "data lineage" to store large working data sets.

It is reported that complex tasks, interactive queries and online processing need a calculation is not available in Hadoop MapReduce, that is, efficient data sharing capabilities, spark with memory data sharing. Intel experts believe that the mapreduce deployed in Hadoop is useful, but memory shows an important advantage in real time, and graphics algorithms may be better suited to existing problems. Intel will continue to work with university researchers to implement research results in the production environment.