Spark Enterprise Application era really come?

Since May 30, the Apache Software Foundation announced the release of the open source Platform Spark 1.0, Spark has repeatedly headlines, has been the focus of data experts. But is Spark's business application era really coming?

From the recent Spark Summit in the United States, we are still full of confidence in spark technology. Spark is often considered a real-time processing environment, applied to Hadoop, NoSQL databases, AWS, and relational databases, and can be used as an API for application interfaces, and programmers process data through a common program. The functions of Spark include SQL query engine, machine learning algorithm, graph processing engine and flow data processing engine.

Many Hadoop vendors have added spark to their own hadoop distributions, such as Hortonworks, Cloudera, IBM, MAPR, and pivotal. Hortonworks, a former founder and CTO Eric Baldeschwieler, believes Spark is likely to become a common technology for large data.

Many supporters believe that spark is a necessary complement to Hadoop and that it also takes on a part of the file system. Spark advocates believe that the value of Spark is that no platform can integrate these independent technologies and functions as spark.

MAPR, the CTO of another Hadoop distributor, is also co-founder M.c. Srivas the combination of spark and hadoop with confidence. He thinks Hadoop's common MapReduce language is difficult to get started, is not friendly to technicians, spark can replace MapReduce language. In addition, since Spark is a memory data processing system, the real-time analysis of Hadoop becomes possible.

Srivas said: "Spark and Hadoop are perfectly matched, and the Application Interface (API) is perfect." Also worth mentioning is the memory processing. MapReduce must be running on a traditional hard drive, but spark can run in memory. Memory processing gives Hadoop the ability to analyze real-time, thanks to spark. ”

, in the past, people's focus on spark focused on data integration and providing a simple and unique interface. But for data scientists, data management is not their interest. As a result, spark gradually increased the functionality of data analysis.

Patrick Wendell, a software engineer for Spark technology supplier Databricks, said that the Spark 1.0 version of the Machine Learning Library (MLLIB) contained 15 predefined machine learning kits, with 1.1 versions expected to reach 30. The developer is developing an interface for the R language and may meet you in the 1.3 release. Although Spark has been known as a data management tool, Wendell believes that the core of Spark is the development of these data analysis code base.

Wendell said: "The code base is Spark's future." It is the source of interest and innovation in the Open-source community. We put the treasure on the code base. ”

Shiquan nine beauty in the ointment

Does this mean that companies should start planning their own spark deployments? Companies have to think twice before they do. Although Spark has various advantages, such as individual API interaction, streaming data and batch data processing capabilities, the ability to run advanced analysis and simple reports at the same time, the spark is still flawed.

Srivas that memory computing faces stability problems. Spark has announced a solution to this problem through the resilient distributed dataset, resilient distributed dataset can provide automatic fail-safe devices through parallel data processing.

Baldeschwieler that spark needs to increase the number of data stores, provide a more powerful code-sharing path, increase the speed of best practices sharing, and develop the portable layer of code. This allows the programmer to write one task at a time, which can be executed in more than one data store, eventually generating the R language interface.

Baldeschwieler concludes: "Although Spark is still a lot of flaws, I still think that Apache Spark is the most exciting technology in the big data age." ”