Mapreduce programming series twelve reduce stage internal details and Adjustment Parameters

Reduce computing is divided into several stages 1. Copy (or shuffle) stage and merge stage parallel

Previously, map results are stored on a local disk. In this case, data needs to be copied from the reduce node from the map node. Store the data in the memory. If the memory size is large, the data is written to the local disk.

At the same time, there are two threads that perform merge operations on the acquired data in the memory and on the disk.

Details:

Call RPC to query the list of map tasks completed by the task tracker. Shuffle is a shuffling operation on all the task tracker hosts, which can disrupt the order of copy data, prevents network Hotspots (a large number of processes read data from a task tracker node ).

Tasks that can be copied are stored in scheduledcopies.

Once data is copied, multiple mapoutputcopier threads are started to copy data through http get requests. If the data is large, the data is saved to the disk. Otherwise, the data is saved to the cache.

In the cache, The inmemoryfsmerge thread is responsible for merge, and the localfsmerger thread is responsible for merge for files.

Therefore, after observing jobtracker, we can see that the map operation has not been completely completed, and the reduce operation has started, that is, it enters the copy stage.

2. parallelism between the sort stage and reduce function that calls CER

Sort sorts the <key, value> data sent from the map stage by merging keys to generate <key, values>

Reduce compiled by the user transfers the preceding <key, values> to the ruduce function for processing.

Parallel algorithms improve program performance. The specific algorithms will be discussed later.

3. Write

Write the result to HDFS.

Reduce optimization parameters

Mapred. Reduce. Parallel. Copies is 5 by default, indicating how many concurrent threads are there to copy data from the task tracker Node

Io. Sort. factor appears again. The default value is 10. It still refers to the number of parallel merged files.

Mapred. Job. Shuffle. Merge. percent is 0.66 by default. If the value exceeds 66%, the data is merged and then written to the disk.

Mapred. inmem. Merge. threshold is 1000 by default. When the threshold value is exceeded, data is merged and written to the disk.

Mapreduce programming series twelve reduce stage internal details and Adjustment Parameters