Mapreduce Execution Process Analysis (based on hadoop2.4) -- (3)

4.4 introduction to reduce class 4.4.1 reduce

After completing the map, the next step is reduce. Yarnchild. Main ()-> reducetask. Run (). The reducetask. Run method starts to be similar to maptask, including initialize () initialization. It depends on whether runjobcleanuptask () and runtaskcleanuptask () are called. After that, I started my work in three steps: Copy, sort, and reduce.

4.4.2 copy

Copy is to obtain the map output file from the nodes that execute each map task. This is the responsibility of the reducetask. reducecopier class. The reducecopier object is responsible for copying the map function output to the machine where the reduce operation is located. If the size exceeds a certain threshold, it is written to the disk. Otherwise, it is stored in the memory. when data is remotely copied, the reduce task starts two background threads to merge files on the memory and disk, prevent excessive memory usage and excessive disk files.

Step 1:

First, in the run method of reducetask, configure mapreduce. Job. Reduce. Shuffle. Consumer. plugin. Class to assemble the plug-in of shuffle. The default implementation is the shuffle class:

1     Class<? extends ShuffleConsumerPlugin> clazz = job.getClass(MRConfig.SHUFFLE_CONSUMER_PLUGIN, Shuffle.class, ShuffleConsumerPlugin.class); 7     shuffleConsumerPlugin = ReflectionUtils.newInstance(clazz, job);9     LOG.info("Using ShuffleConsumerPlugin: " + shuffleConsumerPlugin);

 

Step 2:

Initialize the plug-in and run the run method to obtain the rawkeyvalueiterator instance.

The steps for running the run method are as follows:

Step2.1:

Quantify the number of reduce events:

1     int eventsPerReducer = Math.max(MIN_EVENTS_TO_FETCH, MAX_RPC_OUTSTANDING_EVENTS / jobConf.getNumReduceTasks());3     int maxEventsToFetch = Math.min(MAX_EVENTS_TO_FETCH, eventsPerReducer);

 

Step2.2:

Generate the thread for obtaining the map completion status and start this thread:

 final EventFetcher<K,V> eventFetcher = new EventFetcher<K,V>(reduceId, umbilical, scheduler, this, maxEventsToFetch);

  eventFetcher.start(); 

 

Obtain the completed map information, such as the host and mapid of the map, and put it in the set <maphost> In shuffleschedulerimpl to facilitate the following data copy and transmission.

1       URI u = getBaseURI(reduceId, event.getTaskTrackerHttp()); 3       addKnownMapOutput(u.getHost() + ":" + u.getPort(), 5           u.toString(), 7           event.getTaskAttemptId()); 9       maxMapRuntime = Math.max(maxMapRuntime, event.getTaskRunTime());

 

Step2.3:

In the shuffle class, start the initialization fetcher thread group and start:

 1     boolean isLocal = localMapFiles != null; 2  3     final int numFetchers = isLocal ? 1 : 4  5       jobConf.getInt(MRJobConfig.SHUFFLE_PARALLEL_COPIES, 5); 6  7     Fetcher<K,V>[] fetchers = new Fetcher[numFetchers]; 8  9     if (isLocal) {10 11       fetchers[0] = new LocalFetcher<K, V>(jobConf, reduceId, scheduler,12 13           merger, reporter, metrics, this, reduceTask.getShuffleSecret(),14 15           localMapFiles);16 17       fetchers[0].start();18 19     } else {20 21       for (int i=0; i < numFetchers; ++i) {22 23         fetchers[i] = new Fetcher<K,V>(jobConf, reduceId, scheduler, merger,24 25                                        reporter, metrics, this,26 27                                        reduceTask.getShuffleSecret());28 29         fetchers[i].start();30 31       }32 33     }

 

The run method of the thread is to remotely copy data:

 1     try {  3           // If merge is on, block  5           merger.waitForResource();  8  9           // Get a host to shuffle from11           host = scheduler.getHost(); 13           metrics.threadBusy(); 17           // Shuffle 19           copyFromHost(host); 21         } finally { 23           if (host != null) { 25             scheduler.freeHost(host); 27             metrics.threadFree();  29           } 31         }

 

Step2.4:

Let's take a look at the copyfromhost method. Httpurlconnection is used to transmit remote data.

After a connection is established, data is read from the received stream. Read a map file each time.

1     TaskAttemptID[] failedTasks = null;2 3       while (!remaining.isEmpty() && failedTasks == null) {4 5         failedTasks = copyMapOutput(host, input, remaining);6 7       }

 

In the copymapoutput method above, read a mapid each time and check whether the map output exceeds the size configured by mapreduce. Reduce. Memory. totalbytes according to the Reserve Function in mergemanagerimpl.

Is the value configured in maxmemory * mapreduce. Reduce. Shuffle. Input. Buffer. percent of the current runtime. The default value of buffer. percent is 0.90.

If mapoutput exceeds the configured size, an ondiskmapoutput instance is generated. In the next operation, map output is written to the local temporary file.

If the value does not exceed this value, an inmemorymapoutput instance is generated. In the next operation, the map output is directly written to the memory.

Finally, run shuffleschedded. copysucceeded to copy the file, call the mapout. Commit function, update the status, or trigger the merge operation.

Step2.5:

After all the copies above are completed, close the related threads.

 1    eventFetcher.shutDown();    2  3     // Stop the map-output fetcher threads 4     for (Fetcher<K,V> fetcher : fetchers) { 5       fetcher.shutDown(); 6     }    7  8     // stop the scheduler 9     scheduler.close(); 10 11     copyPhase.complete(); // copy is already complete12     taskStatus.setPhase(TaskStatus.Phase.SORT);13     reduceTask.statusUpdate(umbilical);

 

Step2.6:

Execute the final merge operation, completed by mergemanager in shuffle:

 1 public RawKeyValueIterator close() throws Throwable { 2  3     // Wait for on-going merges to complete 4  5     if (memToMemMerger != null) { 6  7       memToMemMerger.close(); 8  9     }10 11     inMemoryMerger.close();12 13     onDiskMerger.close();14 15    16 17     List<InMemoryMapOutput<K, V>> memory =18 19       new ArrayList<InMemoryMapOutput<K, V>>(inMemoryMergedMapOutputs);20 21     inMemoryMergedMapOutputs.clear();22 23     memory.addAll(inMemoryMapOutputs);24 25     inMemoryMapOutputs.clear();26 27     List<CompressAwarePath> disk = new ArrayList<CompressAwarePath>(onDiskMapOutputs);28 29     onDiskMapOutputs.clear();30 31     return finalMerge(jobConf, rfs, memory, disk);32 33   }

 

Step 3:

Release resources.

mapOutputFilesOnDisk.clear();

 

Copy is complete.

4.4.3 sort

In fact, sort (equivalent to merging) is equivalent to a continuation of sorting. It will be executed after all the files are copied. Use the tool-type merger to merge all files. After this process, a new file will be generated that combines all (not accurate) map task output files, and the map task output files from other servers will be deleted. Determine the sort method to call based on whether hadoop is distributed.

This operation is triggered after step in section 4.3.2 above.

4.4.4 reduce

After completing the preceding steps, return to the run method in reducetask and continue to execute it. Call runnewreducer. Create CER:

1 org.apache.hadoop.mapreduce.Reducer<INKEY,INVALUE,OUTKEY,OUTVALUE> reducer =2 3       (org.apache.hadoop.mapreduce.Reducer<INKEY,INVALUE,OUTKEY,OUTVALUE>)4 5         ReflectionUtils.newInstance(taskContext.getReducerClass(), job);

 

And run the run method. This run method is the run method in org. Apache. hadoop. mapreduce. Cer.

 1 public void run(Context context) throws IOException, InterruptedException { 2  3     setup(context); 4  5     try { 6  7       while (context.nextKey()) { 8  9         reduce(context.getCurrentKey(), context.getValues(), context);10 11         // If a back up store is used, reset it12 13         Iterator<VALUEIN> iter = context.getValues().iterator();14 15         if(iter instanceof ReduceContext.ValueIterator) {16 17           ((ReduceContext.ValueIterator<VALUEIN>)iter).resetBackupStore();       18 19         }20 21       }22 23     } finally {24 25       cleanup(context);26 27     }28 29   }30 31 }

 

The while loop condition is performancecontext. nextkey () is true. This method is implemented in reducecontext. The purpose of this method is to process the next unique key, because the input data of the reduce method is grouped, therefore, each time a key and all values corresponding to the key are processed, and the output of all MAP tasks has been copied and sorted, therefore, Kv pairs with the same key are all adjacent.

In the nextkey method, the nextkeyvalue method is called to obtain the next key value. If no data is available, false is returned. If there is still data, true is returned. Prevent repeated data from being retrieved.

The next step is to call the custom reduce method.

 1 public void reduce(Text key, Iterable<IntWritable> values, 2  3                        Context context 4  5                        ) throws IOException, InterruptedException { 6  7       int sum = 0; 8  9       for (IntWritable val : values) {10 11         sum += val.get();12 13       }14 15       result.set(sum);16 17       context.write(key, result);18 19     }