The principles and differences between MapReduce and spark

Mapreduce and Spark are the two core of data processing layer, understand and learn big data must focus on the link, according to their own experience and everyone to do the knowledge sharing.

650) this.width=650; "Src=" Http://s5.51cto.com/wyfs02/M00/8B/2B/wKioL1hGbEiSjW3wAAEP-Bn8CcE114.jpg-wh_500x0-wm_3 -wmp_4-s_2651010867.jpg "title=" 11111.jpg "alt=" Wkiol1hgbeisjw3waaep-bn8cce114.jpg-wh_50 "/>

First Look atMapreduce, its two most essential process isMapand theReduce,Mapapplication is that we need a mapping transformation of the data-to-one element, such as interception, filtering, or any conversion operation, which is called aMap;Reduceis mainly the aggregation of elements, that is, multiple elements of an element of the aggregation, such as seekingSumwait, this isReduce

Mapreduce Yes appear slowly replace > Is it still being used? Because there are many existing applications also depend on it, it is not an independent existence, has become an irreplaceable part of other ecological, such as pig ET.

while MapReduce greatly simplifies big data analytics, users are increasingly demanding as big data needs and usage patterns grow:

1. more complex multiprocessing requirements, such as iterative calculations , ML, Graph );

2. Low-Latency interactive query requirements (such as Ad-hoc Query )

and MapReduce the architecture of the computational model causes the two types of applications to be inherently slow, and users urgently need a faster computational model to complement MapReduce of congenital defects.

Spark make up for these shortcomings, we come to understand some Spark Advantages of:

1. Each job independent scheduling, can be all the job to do a graph to schedule, the various activities depend on each other, in the scheduling process together, fast.

2. All processes are memory-based, so Spark is often referred to as an iterative, memory-based computing framework.

3.spark provides a richer operator to make the operation more convenient.

4. easier API: Support for Python,Scala and Java

actuallySparkIt can also be implemented insideMapreduce, but here it's not an algorithm, it just providesMapStages andReducestage, but provides many algorithms in two phases. AsMapStage ofmap, FlatMap, filter, Keyby,ReduceStage ofReducebykey, Sortbykey, mean, gourpby, sortand so on.

The above is and we do a knowledge sharing, just a few personal views, for the specific concept of knowledge we can learn privately, there is time to pay attention to " Big Data cn "and" Big Data Times Learning Center "These public service numbers, the introduction of some of the big data trends, big data knowledge and sharing some of the information, are very good, suggest to look at, expect everyone can progress!

This article from "11872756" blog, declined reprint!

The principles and differences between MapReduce and spark