Hadoop-mapreduce Summary 1

Prerequisites:

1. In the inputformat <K, V> interface, there are two methods: inputsplit [] getsplits ();

Recordreader <K, V> getrecordreader ();

2. mapreduce job submission and initialization process.

Job submission:

(1) Command Line submission ....

(2) Get the job ID,

Create an HDFS directory (the directory you specified to store the results)

Upload files to HDFS (Application jar packages, XML files, etc)

Generate the split file (that is, call getsplits in the specified inputformat)

(3) Submit a job to jobtracker.

Job initialization:

The scheduler calls jobtracker. initjob () to construct map tasks and reduce tasks and

Initialization.

3. task running process analysis

Overall map task process:

Read: Use recodreader (getrecordreader () in the inputformat you specify)

In inputsplit, key/value pairs are parsed.

Map: Hand over the key/value in the previous step to map () for processing to form a new key/value.

Collect: In map (), call partition. getpartition () to form a triple <key, value, partition> and write it into the memory ring buffer.

Spill: (1) when the memory buffer is full, sort the data in the buffer (first by partition, then by key)

(2) If a combiner exists, the data is aggregated once.

(3) Writing temporary files

Combine: Merge temporary files (in partitions and sorted)

Overall ruduce task process:

Shuffle: copy a piece of data from each map task (by partition)

Merge: Merge files

Sort: sort. When input data of the reduce () function is aggregated by key,

Reduce:

A map (which class is used here) processes a shard. The map () function processes a pair

Key/value.

This article from the "thick and thin hair" blog, please be sure to keep this source http://duanzhenyue.blog.51cto.com/9360091/1553372

Hadoop-mapreduce Summary 1