Understanding cloud computing from another perspective (2)

Next

3. combiner operation

After completing the map operation, we will summarize the following: a large file, divided into split1 ~ 5, corresponding to map1 ~ 5. Each map processes one split, and each row of each split is processed using the map method of each map. After the above operation, five intermediate results are output.

For each of the five intermediate results, there are many rows, each row is in the key value format, such as Hello 1. In this way, in order to reduce the amount of data transmitted before being passed to the reducer, you can add a combiner process to simplify each intermediate result. Because each intermediate result may have rows with the same key value, such as Hello 1 and hello 1, which can be merged into a row through combiner: Hello 2.

Remember: combiner is part of a Mapper task, followed by the map method. It looks like a reduce operation.

Then, you can pass the combiner output to Cer CER, instead of map output to reducer. The combiner must be specified by yourself. Of course, you can leave it empty.

4. Cer CER operation

Reduce operations are divided into three processes.

(1) shuffle

The previous map or combiner output has a total of five intermediate results, each of which can be divided into (pre-defined r copies) according to the key value ), operations by r reduce (Completed through the partitioner Interface). ThereforeFor r reduce, a piece of data must be obtained from five intermediate results.,Each of the five intermediate results is operated by r reduce.

To put it simply: for one of the performance1 ~ In map5, copy the desired copy to the local device, and then merge for sort.

(2) sort

For each reduce, the incoming data is one of five intermediate results, that is, the shuffle result, which may contain duplicate keys. Therefore, before being input as a reduce, an sort operation will be performed, integrate the same key, for example,

(3) Reduce

Here is the place where we write the program. A total of R reduce can generate R intermediate results.

Note: shuffle, sort, and reduce are completed on reduce nodes, rather than on map nodes. Some books are wrong!

I have talked so much about it. Let's take a look at the figure I just drew.

======================================

Recently, I helped my friends get a public number. at 06:30 every morning, I sent a 60-second speech. If you are interested, please take a look.

No.: wailixs)

QR code:

 

Understanding cloud computing from another perspective (2)