Hadoop Learning (7) WordCount + Block + Split + Shuffle + Map + Reduce

Pure dry goods: WourdCount program example: describes in detail the difference between MapReduce Block + Split + Shuffle + Map + Reduce and the data processing process.

The Shuffle process is the core of MapReduce and focuses on the most critical part of the MR process. To understand MR, Shuffle must be understood. Understanding the Shuffle process will help us better tune the performance of MapReduce jobs and further deepen our understanding of the internal MR mechanism. What is Shuffle? I am referring to a blog of Daniel two years ago. In my article on the MR series, I learned when my predecessors started their work. I really admire it. Here, we will sort out the concepts of our predecessors and add our own opinions to try to find out what the Shuffle process is, what the block process is, and what the split is. Here we will unveil the mystery of MR.

In the previous blog, I briefly introduced the concept of Shuffle. I mentioned split but didn't talk about block. Before learning about Shuffle, we must first understand block and split. Shuffle defines "copy" and "copy" as a piece of data, which can be understood as a split data. However, when data is uploaded to HDFS, the data is divided into blocks, which leads to what is block, what is split, and what is the difference between split and block? After block and split are solved, the map and reduce processes in the center are fixed. Let's get started.

1. Block:

When you upload a file to HDFS, the first step is Data Division. This is a real physical division. After a data file is uploaded to HDFS, it must be divided into one file, the size of each piece can be divided by configuration options in the hadoop-default.xml. By default, each block is 64 MB. A file is divided into multiple 64 MB small files, and the last one may be 64 MB. Note: 64 MB is the default value and can be changed. The following describes how to change it.

<property>     <name>dfs.block.size</name>     <value>67108864</value>     <description>The default block size for new files.</description></property> 

Data is divided into redundancy. Where is the concept of redundancy? To ensure data security, the uploaded files are copied into three copies. When one copy of the data goes down, the other parts can be instantly added. Of course, this is only the default value.

<property>     <name>dfs.replication</name>     <value>3</value>     <description>     Default block replication.The actual number of replications can be specified      when the file is created.The default is used if replication is not specified in create time.     </description></property>

2. Split blocks:

Hadoop has another Data Division. An InputFormat interface is defined here. One of the methods is the getSplits method. Here we will talk about split. As the Hadoop version is updated quickly, the division of split in different versions is completed by different job tasks. Early versions of split have been bent by the JobTracker end, and later versions are completed by the JobClient. After the JobClient is divided, split. file is written to HDFS. When the JobTracker side reads this file, it will know how split is divided. This data division is actually a logical division to make Map tasks better acquire data.

For example:

File1：Block11，Block12，Block13，Block14，Block15File2：Block21，Block22，Block23

If you specify the number of map tasks in the program, for example, 2, and the default value of maptasks is 1 if you do not specify the number of maptasks, In the getSplits method of FileInputFormat (the most common InputFormat implementation, the totalSize = 8 will be calculated first (defined in the source code. Note that the unit of the getSplits function is the number of blocks rather than the number of bytes. The variable "bytesremaining" is followed to indicate the number of remaining blocks, do not calculate the number of bytes according to the variable name, and then calculate goalSize = totalSize/numSplits = 4. For File1, calculate how many blocks a Split has.

long splitSize = computeSplitSize(goalSize, minSize, blockSize);protected long computeSplitSize(long goalSize, long minSize, long blockSize) {   return Math.max(minSize, Math.min(goalSize, blockSize));}

Here, minSize is 1 (indicating that one Split contains at least one Block, and there is no case that one Split contains only a few blocks at zero). The calculated splitSize = 4, therefore, the Split is divided as follows:

Split 1: Block11, Block12, Block13,Block14Split 2: Block15Split 3: Block21, Block22, Block23

What should I do if the number of maps specified by the user is 2 and three splits appear? In JobInProgress, the number of maptasks is specified based on the Splits length. Therefore, the number of map specified by the user is only a reference. See the code in:

　　JobInProgress: initTasks() try {   　　splits = JobClient.readSplitFile(splitFile);  　　} 　　finally {   　　splitFile.close();  　　}  numMapTasks = splits.length;  maps = new TaskInProgress[numMapTasks];

Therefore, the problem is very clear. If you specify 20 map jobs, there will be eight splits (one Block for each Split) at the end, so there will actually be eight MapTasks at the end, that is to say, the number of maptasks is determined by the splits length.

Several simple conclusions:
1) An Integer Block whose split is greater than or equal to 1
2) A split does not contain blocks of two files and does not span the File boundary.
3) The relationship between split and Block is one-to-multiple. The default value is one-to-one.
4) The number of maptasks depends on the splits length.

Another note: In the FileSplit class, there is a private String [] hosts; it seems to indicate the machines on which the FileSplit is placed, in fact, hosts only stores a list of redundant machines in a Block. For example, in the preceding example, Split 1: Block11, Block12, Block13, and Block14, the hosts in the FileSplit finally stores the list of Block11 itself and its redundant machines, that is, Block12, Block13, block14 these blocks are not recorded in FileSplit and contain the relationship between the Block and Split.
This attribute in FileSplit is conducive to the local data issue during Job Scheduling (local data: This attribute is useful for MapReduce Performance Tuning ). If a tasktracker comes to request a task, jobtracker finds a task and finds a maptask. Check whether the hosts in the input FileSplit of this task contains the machine where tasktracker is located, that is, to determine whether the datanode on a machine has a backup of a Block in FileSplit at the same time as the tasktracker.

The connection and difference between Block and Split are briefly described. In fact, the problem is not here. There are new explanations on the Internet, such as: how to deal with cross-row Block and UnputSplit in MapReduce. In fact, we basically want to talk about the difference between the life Block and the Split, which is not described here.

Back to today's focus: Shuffle

3. Shuffle process:

Shuffle, also called Copy stage. The Reduce Task remotely copies a piece of data from each Map Task and writes the data to the disk if the size exceeds the threshold. Otherwise, the data is directly stored in the memory.This is actually a disruption, for example, the random disruption of the element order in the List parameter in the java API Collections. shuffle (list); method.

The official description of the Shuffle process is vague. I suggest you stop reading it. Let's take a look at how the cool guys summarize the Shuffle process. For Shuffle, we need to know how Shuffle effectively transmits the output results of map tasks to the reduce end. In fact, Shuffle describes the process from map task output to reduce task input.

In a cluster environment such as Hadoop, most map tasks and reduce tasks are executed on different nodes. Of course, in many cases, the result of map tasks on other nodes must be pulled across nodes during Reduce execution. If the cluster is running many jobs, the normal execution of tasks will seriously consume network resources in the cluster. This kind of network consumption is normal and we can't limit it. What we can do is to minimize unnecessary consumption. In addition, in the node, disk IO has a considerable impact on the job completion time compared to the memory. From the basic requirements, our expectations for the Shuffle process can be:

1) pull data from the map task end to the Reduce end completely.

2) When pulling data across nodes, minimize unnecessary bandwidth consumption.

3) reduce the impact of disk IO on task execution.

Here, you can think about it.Design the Shuffle process by yourself. What is your design goal?.The optimization mainly lies in reducing the amount of data pulled and using the memory instead of the disk as much as possible.
Here, the Shuffle process specified by the blogger in the source code of Hadoop0.21.0 is used. Take WordCount as an example. Assume that it has eight map tasks and three reduce tasks. The Shuffle process spans the two ends of map and reduce.

3.1 Map stage:

 

Is the running status of a map task. The figure clearly shows the stage at which partition, sort, and combiner are used. From this figure, we can clearly understand the whole process from map data output to the map end where all data is quasi-good.

The entire process is divided into four steps. To put it simply, each map task has a memory buffer that stores the map output results, when the buffer zone is full, the data in the buffer zone needs to be stored in the disk as a temporary file. When the entire map task ends, all the temporary files generated by the map task on the disk are merged, generate the final formal output file and wait for the reduce task to pull the data.

In fact, each step contains multiple steps and details:

1. When a map task is executed, its input data comes from the HDFS block. Of course, in the MapReduce concept, map task only reads the split. The relationship between Split and block is clearly described above. In the WordCount example, assume that the input data of map is a string like "aaa.
2. After the mapper operation, we know that the mapper output is such a key/value Pair: The key is "aaa", and the value is 1. Because the current map end only performs the Add 1 operation, the result set is merged in the reduce task. We know that this job has three reduce tasks. which reduce does the current "aaa" need to be implemented.
MapReduce provides the Partitioner interface, which is used to determine which reduce task to process the output data based on the number of keys, values, and reduce. By default, key hash is followed by the number of reduce tasks. The default modulo mode is only for the average reduce processing capability. If you have requirements for Partitioner, you can customize it and set it to the job.
In the above example, "aaa" returns 0 after passing through Partitioner, that is, the value of this pair should be handled by the first reducer. Next, you need to write data into the memory buffer. The buffer is used to collect map results in batches to reduce the impact of disk IO. The results of our key/value pairs and Partition will be written to the buffer zone. Before writing, the key and value values are serialized into byte arrays. The entire memory buffer is a byte array.
3. This memory buffer has a size limit. The default value is 100 MB. When the map task outputs a lot of results, the memory may burst, so you need to temporarily write data in the buffer to the disk under certain conditions, and then reuse this buffer.This process of writing data from memory to disk is called Spill..This overwrite is completed by a separate thread and does not affect the thread that writes the map result to the buffer zone.. The overwrite thread should not stop map output when it starts, so the entire buffer zone has an overwrite ratio of spill. percent. This ratio is 0.8 by default, that is, when the buffer data reaches the threshold (buffer size * spill percent = 100 MB * 0.8 = 80 MB), The overwrite thread starts, lock the 80 MB memory and execute the overflow write process. The output result of the Map task can also be written to the remaining 20 MB memory, which does not affect each other.
When the overflow write thread starts, Sort the keys in the 80 Mb space (Sort ).Sorting is the default behavior of the MapReduce model.Sorting is also the sorting of serialized bytes. 
Here we can think about it, because the output of map tasks needs to be sent to different reduce ends, while the memory buffer does not merge the data sent to the same reduce end, therefore, such merging should be reflected in disk files. From the official figure, we can also see that the overflow Files written to the disk are merged for different reduce-end values. SoAn important part of the overwrite process is that if many key/value pairs need to be sent to a reduce end, these key/value values need to be spliced into one piece, reduce index records related to partition.
When merging data for each reduce end, some data may be like "aaa"/1, "aaa"/1.In the Wordcount example, the number of times a word appears is simply counted. If there are many keys in the results of the same map task that appear multiple times like "aaa, we should merge their values into one piece. This process is also called reduce.However, in MapReduce terminology, reduce only refers to the process in which the reduce end obtains data from multiple map tasks for computing. In addition to reduce, data can only be combined in an informal manner. As you know, MapReduce treats Combiner as CER Cer.
If the client has set Combiner, it is time to use Combiner.Add the values of key/value pairs with the same key to reduce the amount of data that overflows the disk.. Combiner optimizes the intermediate results of MapReduce, so it is used multiple times throughout the Model. In which scenarios can Combiner be used? Analyze from here,The output of the Combiner is the input of the Reducer. The Combiner must not change the final calculation result.. So from my perspective, Combiner should only be used in scenarios where the input key/value type of Reduce is exactly the same as that of the output key/value type and does not affect the final result. For example, accumulate and maximum.The use of Combiner must be careful. If it is used well, it helps the job execution efficiency, and vice versa, it will affect the final result of reduce.
4,Each write overflow generates an overwrite file on the disk.If the output result of the map is really large and such overflow occurs many times, multiple overflow files will exist on the disk.When the map task is completed, all the data in the memory buffer is written to the disk to form an overflow file.At least one such overwrite file exists in the final disk (if the map output result is very small, only one overwrite file will be generated when the map execution is complete ), because there is only one final file, we need to Merge these overflow files together. This process is called Merge. What is Merge like? As in the previous example, "aaa" reads 5 values from a map task and 8 values from another map. Because they have the same key, merge must be converted into a group. What is group. For "aaa" is like this: {"aaa", [5, 8, 2,…]}, The values in the array are read from different overflow files, and then add these values.Note that because merge combines multiple overflow files into one file, the same key may exist. If the client sets Combiner during this process, combiner is also used to merge the same key.
At this point, all the work on the map end has been completed, and the final generated file is also stored in a local directory that TaskTracker can obtain. Each reduce task continuously obtains information about whether the map task is completed from JobTracker through RPC. If the reduce task is notified, the map task on a TaskTracker is completed, the second half of Shuffle starts.

3.2. Reduce stage:

Simply put, rThe educe task is to constantly pull the final results of each map task in the current job, and then perform merge on the Data pulled from different places, finally, a file is formed as the input file of reduce task.

 

For example, in the detailed map, the Shuffle process on the reduce end can also be summarized by the three points marked in the figure. The premise of current reduce copy data is that it needs to obtain from JobTracker which map tasks have been executed and completed. This process is not detailed in detail. Before the Reducer is actually running, all the time is pulling data, doing merge, and constantly repeating. As in the previous method, I will describe the Shuffle details of the reduce end in segments as follows:
1,In the Copy process, data is pulled simply.The Reduce process starts some data copy threads (Fetcher) and requests the TaskTracker of the map task to obtain the output file of the map task through HTTP. Because the map task has already ended, these files are managed by TaskTracker on the local disk.
2,Merge stage.Here, the merge action is like the merge action on the map side, but the values stored in the array are the copy values of different map terminals. The copied data is first put into the memory buffer. The buffer size here is more flexible than that on the map end. It is set based on the JVM heap size because the Reducer does not run in the Shuffle stage, therefore, the vast majority of memory should be used for Shuffle. Here, merge has three shapes: 1) memory to memory 2) memory to disk 3) disk to disk. The first mode is disabled by default. When the data volume in the memory reaches a certain threshold, the merge from the memory to the disk is started. Similar to the map end, this is also an overwrite process. If you set a Combiner in this process, it will also be enabled, and a large number of overwrite files will be generated on the disk. The second mode of merge is running until the data on the map end ends. Then, the third mode of Disk-to-disk merge is started to generate the final file.
3,Reducer input file.After merge continues, a "final file" will be generated ". Why quotation marks? This file may exist on the disk or in the memory. For us, of course we want it to be stored in the memory and directly used as the CER input, but by default, this file is stored in the disk. As for how to make this file appear in the memory, let's talk about it later. When the Reducer input file is set, the entire Shuffle ends. Then execute Reducer and put the result on HDFS.

The above is the process of Shuffle in MapReduce. From uploading a file to HDFS, we combed the process of file Block, Split read Block, Map, and Reduce. The above content mainly refers to Daniel's blog, which is played by hand, including the map/reduce process.I. It is to understand the internal mechanism of the entire process and deepen understanding. 2. It is to show respect for the in-depth insights of bloggers two years ago. I hope this blog will provide guidance for later learners and deepen their understanding of the internal mechanism process of MapReduce.

Thanks to the blogger: http://langyu.iteye.com/blog/992916

CopyrightBUAA