Preliminary understanding of the architecture and principles of MapReduce

Catalogue

1. MapReduce definition

2. Source of MapReduce

3. MapReduce Features

4. Examples of MapReduce

5. MapReduce Programming Model

6. MapReduce Internal Logic

7. MapReduce Architecture

8. The fault tolerance of the MapReduce framework

9. MapReduce Resources Organization method

1. MapReduce definition

　The MapReduce in Hadoop is a simple software framework based on the applications it writes out to run on a large cluster of thousands of machines and to process terabytes of data in parallel with a reliable fault tolerance

2. Source of MapReduce

Hadoop MapReduce originated in a mapreduce paper published by Google in December 2004. Hadoop MapReduce is actually　　 a cloned version of Google MapReduce

3. MapReduce Features

Why is MapReduce so popular? Especially now the Internet + ERA, Internet + companies are using MapReduce. The reason why MapReduce is so popular is that it has several features　　 :

　1. MapReduce is easy to program

It simply implements a number of interfaces, it can complete a distributed program, the distributed program can be distributed to a large number of inexpensive PC machine operation. That means you write a distributed program that is identical to writing a simple serial program. It is because of this feature that the MapReduce programming becomes very popular.

　2. Good extensibility

When your computing resources are not met, you can expand the computing power by simply adding machines.

3. High fault tolerance

The purpose of the MapReduce design is to enable the program to be deployed on a cheap PC machine, which requires a high level of fault tolerance. For example, if one of the machines is hung up, it can move the above computing task to a different node, so that the task does not fail, and the process does not require manual involvement, but is done entirely internally by Hadoop.

4. Suitable for off-line processing of petabytes and above massive data

It is suitable for offline processing and is not suitable for online processing. such as a millisecond-level return of a result, MapReduce is difficult to do

MapReduce has a lot of advantages, but it also has a place that is not good at it. Here is not good does not mean that it can not do, but in some scenarios to achieve poor results, not suitable for MapReduce to deal with, mainly in the following aspects:

1. Real-time computing

MapReduce cannot return results in milliseconds or seconds, like Mysql.

2. Flow type calculation

The input data is dynamic when streaming, and the input data set of MapReduce is static and cannot be changed dynamically. This is because the design features of the MapReduce itself determine that the data source must be static.

3. DAG (directed graph) calculation

A dependency exists for multiple applications, and the input of the latter application is the previous output. In this case, MapReduce is not unable to do, but after use, each MapReduce job output will be written to disk, resulting in a lot of disk IO, resulting in very low performance

4. Examples of MapReduce

　To analyze the programming model of MapReduce, here we take WordCount as an example. Just like the starter program in Java, C + + and other programming languages Hello Word, WordCount is the simplest starter program for MapReduce.

Let's step through the analysis.

1, the scene: if there are a large number of files, which are stored in words.

Similar scenarios: Although WordCount is simple, it is a model for many important applications.

1) Search engine, statistics the most popular K search terms.

2) Statistical search term frequency, help to optimize the search word hint.

2, Task: How do we count the number of occurrences of each word?

3, the problem specification is: There are a number of files (scale of TB or petabytes), how to count the occurrences of all the words in these files.

4. Solution: First, the number of occurrences of words in each file is counted, and then the number of occurrences of the same word in different files is accumulated.

5. MapReduce Programming Model

　　From the above analysis, it is actually a typical MapReduce process. Let's analyze the MapReduce process.

　　

　　Process is roughly divided into the following steps:

1, suppose a file has three lines of English words as a MapReduce input, here through the splitting process to divide the file into 3 pieces. After splitting the 3 pieces of data can be processed in parallel, each piece to a map thread processing.

2, each map thread, with each word as key, 1 as the frequency of the number of value, and then output.

3. the output of each map is shuffling (mixed), the same word key is placed in a bucket, and then it is given to reduce processing.

4, reduce to accept the data after the shuffle, will be the same word to merge, get each word frequency, and finally will be good statistics of each word frequency as the output results.

The above is the approximate process of MapReduce, the first two steps can be regarded as the map phase, the latter two steps can be seen as the reduce phase. Let's take a look at how MapReduce is generally implemented

1, Input: First MapReduce input is a series of key/value pairs. The key represents the offset per line, and value represents the word entered for each line.

2, the user provides the map function and the implementation of the reduce function

Map (k,v)--list (K1,V1)

Reduce (k1,list (v1))--(K2,V2)

The map function converts each word to the output of Key/value, where key is each word, and value is frequency 1. (K1,V1) is the intermediate key/value result pair of the map output. Reduce merges all the words of the same word, such as K1 the words, the word frequency as list (v1), and merging them into (K2,V2). After the reduce is combined, the final output of a series of (K2,V2) key-value pairs

Let's take a look at the pseudo-code of MapReduce.

Map (Key,value)://Map function, key represents offset, value represents each line of Word

For every word w in value://loops each row of data, outputting key-value pairs (w,1) of each word and frequency of words

Emit (w,1)

Reduce (Key,values)://Reduce function, key represents a word, value represents all word frequency numbers

Result=0

For the Count V in values://Cycle frequency collection, find the total word frequency of the words, and then output (Key,result)

Result+=v

Emit (Key,result)

In this case, we can summarize the MapReduce: MapReduce divides the entire operational process of the job into two phases: the map phase and the reduce phase.

1. Map Stage

The map phase consists of a number of map tasks. These map tasks can be run at the same time, and each map task is made up of the following three parts.

1) components to parse the input data: InputFormat

Because different data may be stored differently, this requires a InputFormat component to parse the data. By default, it provides a textinputformat to interpret the data. Textinputformat is the text file input format we mentioned earlier, it interprets each line of the file as (Key,value), the key represents the offset per line, and value represents the content of each row of data. Normally we don't need to customize InputFormat, because MapReduce provides many kinds of inputformat implementations, and we can interpret them according to different data formats and choose different InputFormat.

2) input data processing: Mapper

This Mapper must be implemented because the data is handled differently depending on the business.

3) Data group: Partitioner

Mapper after data processing, the output key will be partitioner grouped or split into buckets to select different reduce. By default, Partitioner will hash the key of the map output, for example, there are 6 reduce task, it is modulo (mod) 6, if the key hash value is 0, select the No. 0 Reduce Task, if the key hash value is 1, Select the first Reduce Task. So different map to the same word key, its hash value modulo is the same, so will be handed to the same reduce to deal with.

2. Reduce phase

The reduce phase consists of a certain number of reduce tasks. These reduce tasks can be run at the same time, and each reduce task is made up of the following four parts.

1) Data Transport copy

The Reduce Task copies the results of each map process, reading a subset of the results from each map. The data that each Reduce task copies is determined by the partitioner above.

2) data sorted by key

When the Reduce task finishes reading the data, it is sorted by key. After the key is sorted, the same key is assigned to a group, which is processed by the same reduce task.

3) Data processing: Reducer

Take WordCount as an example, the same word key is divided into a group, the same reducer processing, so that the frequency of each word to achieve the statistics.

4) Data output format: OutputFormat

The results of reducer statistics will be output according to the OutputFormat format. By default, the output format is Textoutputformat, in the case of WordCount, where the key is the word, and value is the frequency of words.

InputFormat, Mapper, Partitioner, reducer, and OutputFormat are all users can achieve. In general, users only need to implement mapper and reducer, and others will use the default implementation.

6. MapReduce Internal Logic

　Below we analyze the data processing process of mapreduce through the internal logic of MapReduce. Let's take wordcount as an example and look at the internal logic of MapReduce, as shown in

　　

　The approximate process for the internal logic of MapReduce is done in the following steps

1. First, the data in HDFS is Split as the input of the MapReduce. As previously mentioned in the article, the data in HDFs is stored in blocks, and how does this turn into split as input? In fact, block is the term in HDFS, and Split is the term in MapReduce. By default, a Split can correspond to a block and, of course, multiple blocks, and the correspondence between them is determined by InputFormat. By default, Textinputformat is used, when a split corresponds to a block. Suppose there are 4 blocks, 4 split, Split0, Split1, Split2, and Split3, respectively. At this time through the InputFormat to read each split inside the data, it will parse the data into one (Key,value), and then to the already written mapper function to deal with.

2. Each mapper will parse the input (key,value) data into words and word frequency, such as (a,1), (b,1) and (c,1) and so on.

3, Mapper parse out the data, such as (a,1), after Partitioner, will know the choice of which reducer to deal with. After each map phase, the data is output to the local disk.

4. In the reduce phase, each reduce is required to shuffle read its corresponding data. After all the data has been read, it is sorted by sort and then handed to Reducer for statistical processing. For example, the first reducer reads two (a,1) key-value pairs of data and then makes a statistical result (a,2).

5, the Reducer processing results, in OutputFormat data format output to the various file paths of HDFS. Here the OutputFormat default is textoutputformat,key for words, value is the word frequency number, and the delimiter between key and value is "\tab". As shown, (a 2) output to Part-0, (b 3) output to Part-1, (c 3) output to Part-2

7. MapReduce Architecture

　　Like HDFs, MapReduce is also a master/slave-based architecture, and its architecture diagram is as follows

　　

　　The MapReduce consists of four components, client, Jobtracker, Tasktracker, and task, which are described in detail in the following four components.

1) Client Clients

Each Job will be stored in HDFS using the client class to package the application and configuration parameters into a JAR file, and submit the path to the Jobtracker Master service, and then the master creates each Task (i.e., maptask and reducetask) to distribute them to the various Tasktracker services to execute.

2) Jobtracker

Jobtracke is responsible for resource monitoring and job scheduling. Jobtracker monitors the health of all tasktracker and jobs, transfers the corresponding tasks to other nodes once the discovery fails, and jobtracker tracks the progress of the task, the amount of resources used, and tells the Task Scheduler The scheduler chooses the appropriate task to use these resources when the resource is idle. In Hadoop, the Task Scheduler is a pluggable module that allows the user to design the appropriate scheduler to suit their needs.

3) Tasktracker

Tasktracker periodically reports the usage of resources on this node and the progress of tasks to jobtracker through heartbeat, and receives the commands sent by Jobtracker and performs the corresponding actions (such as starting new tasks, killing tasks, etc.). Tasktracker uses "slot" to divide the amount of resources on this node equally. "Slot" stands for compute resources (CPU, memory, etc.). Once a task acquires a slot to run, the Hadoop Scheduler is used to assign the free slots on each tasktracker to the task. Slots are divided into the map slot and the reduce slot two, which are used by map task and reduce task respectively. Tasktracker limits the concurrency of a task by the number of slots (configurable parameters).

4) Task

The task is divided into map task and reduce task two, all of which are started by Tasktracker. HDFS stores data in a fixed-size block as the base unit, whereas for MapReduce the processing unit is split. Split is a logical concept that contains only some metadata information, such as the starting position of the data, the length of the data, the node where the data resides, and so on. Its partitioning method is entirely up to the user's own discretion. However, it is important to note how much of split determines the number of map tasks because each split will only be handed to a map task handler. The relationship between Split and block is as shown

　　

　　The map task executes as shown: The map task first parses the corresponding split iteration into a Key/value pair, calls the user-defined map () function, and then stores the temporary result on the local disk. Where temporary data is divided into several partition, each partition will be processed by a reduce Task

　　

　The Reduce Task execution process is shown. The process is divided into three stages:

• Reads the map Task intermediate result (called the "Shuffle phase") from the remote node;

• Sort Key/value Pairs according to key (called "sort Stage");

• Read < Key, value List>, call the user-defined Reduce () function and save the final result to HDFs (known as "Reduce phase")　　

　　

8. The fault tolerance of the MapReduce framework

　One of the biggest features of MapReduce is that there is a good fault tolerance, even if your node hangs out of one, 2, 3, there is no problem, it can run as usual, your job or application to run the completion. No node hangs and your job fails to run. So what kind of mechanism does MapReduce have so that it has such a good fault tolerance? Let's take a look at this in turn.

1, Jobtracker

Unfortunately, Jobtracker has a single point of failure, and the entire cluster is not available once a failure occurs. This is the problem in 1.0, and in 2.0 the problem has been solved. However, it is reassuring that, even in 1.0, MapReduce does not often fail. It may be a year or several failures, after a failure, you restart, and then re-submit the job, it will not be like HDFS data loss. Because MapReduce is a computational framework, the computational process can be reproduced, even if a service is hung up, you restart the service, and then resubmit the job, it will not affect your business.

2, Tasktracker

Tasktracker periodic report to Jobtracker heartbeat, if a certain period of time did not report the heartbeat, Jobtracker think the Tasktracker hung up, it will be all the above tasks scheduled to other Tasktracker (node) run. This will not affect the operation of the entire cluster, even if a node is hung.

3, Maptask and Reducetask

Maptask and Reducetask may also run out. For example, if the memory goes out or the disk hangs, the task hangs. At this point, Tasktracker will return each maptask and Reducetask's running status to Jobtracker,jobtracker. Once a task is found to be dead, it dispatches the task to the other node through the scheduler. This way, even if the task is dead, it will not affect the operation of the application.

9. MapReduce Resources Organization method

　　The MapReduce computing framework does not directly invoke multi-dimensional resources such as CPU and memory, it abstracts multidimensional resources into "slots" and "slots" to describe the number of resources. Administrators can configure the number of slots individually on each node. Slots can be divided into map slots and reduce slots. To some extent, slots can be viewed as "task run parallelism." If a node is configured with 5 map slots, the node runs up to 5 map tasks, and if a node is configured with 3 reduce slots, the node runs up to 3 reduce tasks. Here we describe the MAP slot and the reduce slot separately.

1. Map Slot

• The map slot can be used to run the resources of the map task, and only the map task can be run.

• Each map task typically uses a map slot. For example, like a capacity scheduler, it can have relatively large maptask. This maptask uses more memory, so it may use multiple map slots.

2. Reduce Slot

• The Reduce slot can be used to run Reducetask and can only run Reducetask.

• Each reducetask typically uses a reduce slot. For example, like a capacity scheduler, it can have relatively large reducetask. Such reducetask use more memory, then it may use multiple reduce slots

If you think reading this blog gives you something to gain, you might want to click " recommend " in the lower right corner.
If you want to find my new blog more easily, click on " Follow me " in the lower left corner.
If you are interested in what my blog is talking about, please keep following my follow-up blog, I am " Liu Chao ★ljc".

This article is copyright to the author and the blog Park, Welcome to reprint, but without the consent of the author must retain this paragraph, and in the article page obvious location to the original link, otherwise reserves the right to pursue legal responsibility.

Preliminary understanding of the architecture and principles of MapReduce