MapReduce programming series three Reduce stage implementation, mapreducereduce

MapReduce programming series three Reduce stage implementation, mapreducereduce

Reduce code is used for addition and statistics,

package org.freebird.reducer;import java.io.IOException;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.mapreduce.Reducer.Context;import org.apache.hadoop.mapreduce.Reducer;public class LogReducer<Key> extends Reducer<Key, IntWritable, Key,IntWritable> {    private IntWritable result = new IntWritable();    public void reduce(Key key, Iterable<IntWritable> values,                       Context context) throws IOException, InterruptedException {        int sum = 0;        for (IntWritable val : values) {            sum += val.get();        }        result.set(sum);        context.write(key, result);    }}

Iterate through values and retrieve all values, which are 1, simple addition.

Then the result is written to the context. Note that the context here is the Context of the CER package.

Why is the Mapreduce model divided into two stages: map and reduce? Is it for the fine granularity of task allocation?

To achieve distributed computing and improve computing efficiency.
I personally think that if you want to improve the computing efficiency of a task that needs to process a large number of datasets, you can divide the task into several small parts. Each part processes a part of the data, just like the map task of hadoop, however, in many cases, the entire dataset is required for computing operations. However, splitting a single small part can improve the computing efficiency, but the actual requirements cannot be fulfilled, which makes no sense, therefore, a reduce stage is added to summarize the computing results of multiple parts for processing, so as to better meet general requirements. Of course, this is not omnipotent. In many cases, it still cannot meet actual needs. This is why hadoop is not omnipotent. Many problems cannot be solved using hadoop.

In the entire map/reduce process of hadoop mapreduce, are map and reduce executed on master or slaver respectively?

There are Map and Reduce Execution Code on each slave (datanode.
When a Job is submitted, the configuration files and jar files of the job are packaged and copied.
To the various datanode and perform local execution.