A detailed description of the MapReduce principle in HADOOP1

Analyze MapReduce operation mechanism: principle:

The explanation of the schematic can be divided into the following sections

1, the client submits a MapReduce jar package to Jobclient

2. Jocclient communicates via RPC and Jobtracker, returning an address (HDFS) that holds the jar package

3. Jobclient writes the jar package to HDFs (the address on the PATH=HDFS (this address is returned by the Jobtracker with the second step) +jobid)

Copies the resource files needed to run the job to HDFs, including the jar files packaged by the MapReduce program, configuration files, and client-computed input partitioning information. These files are stored in a folder that is created specifically for the job by Jobtracker. The folder name is the job ID of the activity. The jar file will have 10 copies (Mapred.submit.replication property control) By default, and the input partitioning information tells the Jobtracker how many map tasks should be started for the job.

4. Start submitting tasks (description of tasks: including location of Jobid,jar storage, configuration information, etc.)

Jobclient calling Jobtracker's Submitjob () method to submit a task

5, Jobtracker to initialize the task

Jobtracker will place the submitted job in an internal queue, which is assigned to the job scheduler, and the initialization of the task involves creating an object that represents the running job-encapsulating tasks and logging information to track the status and information of the task.

6, read the files to be processed on the HDFs, start to calculate the input shards, each shard corresponds to a mappertask

When the job scheduler dispatches to the job according to its own scheduling algorithm, a mappertask task is started for each slice based on the input partitioning information.

7, Tasktracker through the heartbeat mechanism to collect tasks (Task description information)

Map tasks are not randomly assigned to a tasktracker, there is a concept called: Data localization (data-local). This means that the map task is assigned to the Tasktracker that contains the map processing. The program jar package is also copied to the Tasktracker to run, which is called "Operation movement, data does not move". When you assign a reduce task, data localization is not considered. Tasktracker sends a heartbeat to jobtracker every once in a while, telling Jobtracker that it is still running, and that the heartbeat also carries a lot of information, such as the progress of the current map task. When Jobtracker receives the last task completion information for the job, it sets the job to "success". When Jobclient queries the state, it learns that the task is complete and displays a message to the user

8. Download the required jar, config file

9. Tasktracker start a Java child subprocess to perform specific tasks (Mappertask or Reducertask)

The execution process of the map function side:

A: Each input shard will have a map task processing, by default, a fast size of HDFs is a shard, the result of the map output is placed in a ring memory buffer (the buffer size defaults to 100M, Controlled by the Io.sort.mb property) when the buffer is about to overflow (default is 80% of the buffer size, by Io.sort. Spill. Percent Property control), an overflow file is created in the local file system that writes the data of the buffer to this file

B: Before writing to disk, the thread first divides the data into the same number of partitions based on the number of reduce tasks, which is the data for one partition for a reduce task, to avoid some of the reduce tasks being allocated to a large amount of data, while some reduce tasks are assigned to very little data. In fact, partitioning is the process of hashing data, then sorting the data in each partition, and if Combiner is set at this point, the sorted result is combiner, and the goal is to have as little data as possible to write to the disk.

C: When the map task output last record, there may be a lot of overflow files, then need to merge these files, the merging process will continue to sort and combiner operations, the purpose is two: 1, to minimize the amount of data written to disk each time, 2, Minimizing the amount of data transmitted over the next replication phase, and finally merging into a partitioned sorted file, to reduce the amount of data transmitted over the network, the data can be refined, as long as the mapred.compress.map.out is set to true.

D: Copy the data from the partition to the corresponding reduce task, and one might ask: How does the data in the partition know which reduce it corresponds to? In fact, the map task has been and its father Tasktracker keep in touch, and Tasktracker has been and jobtracker keep heartbeat. So the macro information in the whole cluster is saved in the Jobtracker. OK, as long as the reduce task gets the corresponding map output location to Jobtracker.

Reduce function-Side Execution process

A:reduce will receive data from different map missions, and the data from each map is ordered, if the amount of data received by the reduce side is quite small, is stored directly in memory (the buffer size is controlled by the Mapred.job.shuffle.input.buffer.percent property, which represents the percentage of heap space used for this purpose), if the amount of data exceeds a certain percentage of the buffer size (by Mapred.job.shuffle.merg E.buffer.percent), the data is merged and then overflowed to disk.

B: With the increase in overflow files, the background thread will merge them into a larger, ordered file, in order to save time for the later merges, in fact, regardless of the map or the reduce side, MapReduce is repeated to perform the sort, merge operation,

C: The merging process produces many intermediate files (written to disk), but Maoreduce will make the data written to disk as small as possible, and the result of the last merge is not written to the disk, but is entered directly into the reduce function.

10. Write the results into HDFs

A detailed description of the MapReduce principle in HADOOP1