Spark VS MapReduce

Apache Spark, a Memory data processing framework, is now a top-level Apache project. This is an important step toward stability for spark, as it is increasingly replacing MapReduce in next-generation big data applications.

MapReduce is interesting and useful, but now it seems that spark is starting to take the reins from it and become the primary processing framework for the new Hadoop workloads. This technology took a very significant step in the last Thursday: the Apache Software Foundation announces that Spark is now a top-level project.

Because it is faster and easier to program than MapReduce, Spark already includes a large number of users and code contributors. This means that it is ideal for applications with next-generation big data that may require lower latency queries, real-time processing, or iterative computing on the same data (ie, machine learning). Spark was founded by the University of California, Berkeley, and has created a company called Databricks to commercialize its operations.

Spark is technically a standalone project, but it's always designed to work with a Hadoop distributed file system. It can run directly on HDFs, and through yarn, it can run with mapreduce jobs on the same cluster. In fact, Hadoop's pioneer Cloudera company now offers enterprise-class support for spark customers.

Spark VS MapReduce