The process of shuffle and sorting in Map, reduce tasks

The process of shuffle and sorting in Map, reduce tasks

Process Analysis:

Map End:

1. Each input shard will have a map task to handle, by default, the size of one block in HDFs (64M by default) is a shard, and of course we can set the size of the block. The result of the map output is temporarily placed in a ring memory buffer (the buffer size defaults to 100M, controlled by the Io.sort.mb property), when the buffer is about to overflow (default is 80% of the buffer size, Controlled by the Io.sort.spill.percent property, an overflow file is created in the local file system and the data in that buffer is written to this file.

2. Before writing to disk, the thread first divides the data into the same number of partitions based on the number of reduce tasks, which is the data for one partition for a reduce task. This is done to avoid some of the reduce tasks being allocated to large amounts of data, while some reduce tasks have little or no data embarrassment. In fact, partitioning is the process of hashing data. The data in each partition is then sorted, and if combiner is set at this point, the sorted result is combia and the purpose is to have as little data as possible to write to the disk.

3. When the map task outputs the last record, there may be a lot of overflow files, and these files need to be merged. The process of merging is continuously sorted and combia, with two purposes: 1. Minimize the amount of data that is written to the disk each time; 2. Minimize the amount of data transmitted over the next replication phase of the network. Finally, it is merged into a partitioned and sorted file. In order to reduce the amount of data transmitted over the network, the data can be compressed, as long as the mapred.compress.map.out is set to true.

4. Copy the data from the partition to the corresponding reduce task. One might ask: How does the data in the partition know which reduce it corresponds to? In fact, the map task has been and its father Tasktracker keep in touch, and Tasktracker has been and jobtracker keep heartbeat. So the macro information in the whole cluster is saved in the Jobtracker. OK, as long as the reduce task gets the corresponding map output location to Jobtracker.

Here, the map end is analyzed.So what is shuffle? Shuffle Chinese means "shuffle", if we look at this: a map generated data, the result of the hash process is allocated to different reduce tasks, is not a data shuffling process?？ Oh.

Reduce side:

1. Reduce receives data from different map missions, and the data from each map is ordered. If the amount of data accepted by the reduce side is quite small, is stored directly in memory (the buffer size is controlled by the Mapred.job.shuffle.input.buffer.percent property, which represents the percentage of heap space used for this purpose), if the amount of data exceeds a certain percentage of the buffer size (by Mapred.job.shuffle.merg E.percent), the data is merged and then overflowed to disk.

2. As overflow files increase, background threads merge them into a larger, ordered file to save time for subsequent merges. In fact, regardless of the map or the reduce side, MapReduce is repeated to perform the sort, merge operation, now finally understand why some people say: sort is the soul of Hadoop.

3. The process of merging produces many intermediate files (written to disk), but MapReduce makes the data written to disk as small as possible, and the result of the last merge is not written to disk, but is entered directly into the reduce function.

The process of shuffle and sorting in Map, reduce tasks