MapReduce Execution Process

First, the basic implementation process

The MapReduce process is divided into two stages: the map function phase and the reduce function phase

(1) The map function is used to filter out non-required data, in the form of key-value pairs output, the key is the file location offset, the value of the data to be analyzed, the core purpose of the Map function is to form the index of the data for the reduce function to facilitate the analysis of the data.

(2) The reduce function takes the output data of the map function as the data source, analyzes the data accordingly, and outputs the result as the final target data.

Ii. distributed execution process in practical application

(1) A map, a reduce

(2) Multiple map nodes, one reduce

(3) Multiple maps, multiple reduce

Implementation efficiency optimization by combiner function

Because the output of the map task is passed to the reduce task, it is the transmission between the nodes, which is the bandwidth, which restricts the maximum throughput of the program execution, in order to reduce the data transfer between map and reduce, The combiner function was added to the map to preprocess the map results, and the Combiner function was run on the map node.

Iv. examples

MapReduce Execution Process