Working Mechanism of Map-Reduce

Mapper

The idea of "map-reduce" is "divide and conquer"

Mapper is responsible for "dividing" the complex tasks into several "simple tasks" for execution.

There are several meanings of "simple tasks": 1. The data or computing scale is greatly reduced compared with the original task;

2. Nearby computation is allocated to the node that stores the required data for computation.

3. These small tasks can be computed in parallel without mutual dependency.

Reduce

? Summarize the results of the map stage
? The number of reducers is determined by the Project mapred. Reduce. Tasks in the mapred-site.xml configuration file. Missing
The saved value is 1, which can be overwritten by the user

 

Shuffler

? A step in the middle of Mapper and reducer (can be left blank)
? You can split and combine the Mapper output into N parts according to a certain key value, and match the key value with a certain range.
The output is sent to a specific CER for processing.
? The CER process can be simplified.

 

Working Mechanism of Map-Reduce