Hadoop programming notes (1): Mapper and reducer class details

Document directory

• 1. Shuffle
• 2. Sort
• 3. Reduce

This series of hadoop programming notes focuses on hadoop programming, including the usage and functions of main classes and interfaces, programming methods, and best practices, if you want to learn more about the features and functions of hadoop and its affiliated ecosystem (such as pig, hive, and hbase), refer to another series of hadoop learning notes. I am well aware of my limited capabilities. I hope you can give me some advice when writing something wrong ~~
Description: This article comes from hadoop1.0.4 API.

1. mapper

Maps are the individual tasks which transform input records into a intermediate records. the transformed intermediate records need not be of the same type as the input records. A given input pair may map to zero or lower output pairs.
A map task is an independent task that converts an input record to an intermediate record. The converted intermediate record does not need to be of the same type as the input record. A given input key-value pair may be mapped to zero or multiple output key-value pairs.

The hadoop map-Reduce framework spawns one map task for eachInputSplitGenerated byInputFormatFor the job.MapperImplementations can accessConfigurationFor the job viaJobContext.getConfiguration().
Inputformat generates one or more inputsplits for the corresponding job, And the hadoop map-Reduce framework generates a map task for each inputsplit. The Mapper implementation class can be passed throughJobcontext. getconfiguration () to obtain the configuration object of the job (which stores the configuration information of various jobs ).

The framework first CILSsetup(org.apache.hadoop.mapreduce.Mapper.Context), Followedmap(Object, Object, Context)For each key/value pair inInputSplit. Finallycleanup(Context)Is called.
The hadoop map-Reduce framework first callsSetup (Org. Apache. hadoop. mapreduce. Mapper. Context) creates a work environment and callsThe map (object, object, context) function processes the input data. After the map task is completed, callCleanup (context) to do some cleanup work. 

All intermediate values associated with a given output key are subsequently grouped by the framework, and passed toReducerTo determine the final output. Users can control the sorting and grouping by specifying two keyRawComparatorClasses.
All values with the same intermediate output key (key) are subsequently grouped by the framework and then transmitted to Cer CER to determine the final output. You can specify two keysRawcomparator class to control the sorting and grouping of intermediate data. 

TheMapperOutputs are partitioned perReducer. Users can control which keys (and hence records) Go to whichReducerBy implementing a customPartitioner.
Mapper output will be partitioned for each reduceer. You can implement a customPartitioner controls which key (and therefore the corresponding record) flows to the reducer.

Users can optionally specifycombiner,Job.setCombinerClass(Class), To perform local aggregation of the intermediate outputs, which helps to cut down the amount of data transferred fromMapperToReducer.
You can alsoJob. setcombinerclass (class) to specify a combiner to execute local merge of intermediate output data, thus reducing the amount of data transmitted from Mapper to Cer.

Applications can specify if and how the intermediate outputs are to be compressed and whichCompressionCodecS are to be used viaConfiguration.
The application can use configuration to specify whether or not the intermediate output data is compressed, and whichCompressioncodec for compression.

If the job has zero reduces then the output ofMapperIs directly written toOutputFormatWithout sorting by keys.
If this job does not have a reduce task, the Mapper output will be directly written to outputformat without sorting by keyword.

In addition to the setup (), map (), and cleanup () methods described above, the er abstract class has another method: Public void.Run(Mapper. Context context). You can reload this method to implement more mapper control.

2. Cer CER

The CER implementation class can be passed throughJobcontext. getconfiguration () to obtain the configuration object of the job (which stores the configuration information of various jobs ).

ReducerThere are three major stages:

1. Shuffle

TheReducerCopies the sorted output from eachMapperUsing HTTP protocol ss the network.
Reducer copies sorted output data from Mapper to the local through the network via HTTP

2. Sort

The framework merge sortsReducerInputskeyS (since differentMapperS may have output the same key ).
The mapreduce framework fuses and sorts CER input by key (because different mappers may output the same key to a reducer)

The shuffle and sort phases occur simultaneously I. e. While outputs are being fetched they are merged.
Shuffle and sort can be performed simultaneously. For example, map output data can be merged during transmission.

2.1 secondarysort

To achieve a secondary sort on the values returned by the value iterator, the application shoshould extend the key with the secondary key and define a grouping comparator. the keys will be sorted using the entire key, but will be grouped using the grouping comparator to decide which keys and values are sent in the same call to reduce. the grouping comparator is specifiedJob.setGroupingComparatorClass(Class). The sort order is controlledJob.setSortComparatorClass(Class).
To obtain the second sorting for the combination of values returned by the value iterator, the application should extend the keyword, that is, use a "second keyword" (secondary key), and then define a group comparator, in this case, the sorting of keywords is determined by the entire keyword group, but the group comparator is used for grouping to determine which key-value pairs are sent to the same CER for processing. Group comparator can useJob. setgroupingcomparatorclass (class ).Job. setsortcomparatorclass (class) control.

For example, say that you want to find duplicate web pages and tag them all with the URL of the "best" known example. You wocould set up the job like:
For example, if you want to find some web pages with repeated content and then assign them all the URLs on the most popular pages, you can set the job as follows:

• Map Input key: URL
• Map input value: Document
• Map output key: Document checksum, URL PageRank
• Map output value: URL
• Partitioner: By checksum
• Outputkeycomparator: By checksum and then decreasing PageRank
• Outputvaluegroupingcomparator: By checksum

Note: PageRank is an important indicator to measure the importance and popularity of a webpage.

3. Reduce

In this phasereduce(Object, Iterable, Context)Method is called for each<key, (collection of values)>In the sorted inputs.
In the reduce stage, each <key, (collection of values)> In the sorted Reduce input is called.reduce(Object, Iterable, Context)

The output of the reduce task is typically written toRecordWriterViaTaskInputOutputContext.write(Object, Object).
Normally, the output of the reduce task passes throughTaskinputoutputcontext. Write (object, object) toIn recordwriter

The output ofReducerIsNot re-sorted.
The CER output is not reordered.

This article is original, reproduced please indicate the source: http://www.cnblogs.com/beanmoon/archive/2012/12/06/2804594.html