Mapreduce-partition analysis (transfer)

Http://blog.oddfoo.net/2011/04/17/mapreduce-partition%E5%88%86%E6%9E%90-2/

Location of Partition

Partition location

Partition is mainly used to send the map results to the corresponding reduce. This has two requirements for partition:

1) balance the load and distribute the work evenly to different reduce workers as much as possible.

2) Efficiency and fast allocation speed.

Partitioner provided by mapreduce

The default partitioner of mapreduce is hashpartitioner. In addition to this mapreduce, three types of partitioner are provided. As shown in:

Patition Class Structure

1. partitioner is the base class of partitioner. If you need to customize partitioner, you also need to inherit this class.

2. hashpartitioner is the default partitioner of mapreduce. The calculation method is

Which reducer = (key. hashcode () & integer. max_value) % numreducetasks to obtain the current reducer.

3. binarypatitioner inherits from partitioner <binarycomparable, V>, and is a special subclass of partitioner. This class provides leftoffset and rightoffset. When calculating the which reducer, only hash is used for the range [rightoffset, leftoffset] of the key value K.

Which reducer = (hash & integer. max_value) % numreducetasks

4. keyfieldbasedpartitioner is also a hash-based partitioner. Unlike binarypatitioner, it provides multiple intervals for hash calculation. When the number of intervals is 0, keyfieldbasedpartitioner degrades to hashpartitioner.

5. The totalorderpartitioner class can sort the output in full order. Unlike the preceding three partitioner, this class is not based on hash. In the next section, we will introduce totalorderpartitioner in detail.

Totalorderpartitioner

The output of each reducer is ordered by default, but the inputs between reducers are disordered. Totalorderpartitioner is used if the output is in full sorting.

To use totalorderpartitioner, you must provide a partition file for totalorderpartitioner. This file requires that the number of keys (these keys are called the Division) be the same as the number of current reducers-1 and are arranged in ascending order. We will still mention the reason why such a file is used and the details of the file.

Totalorderpartitioner provides two solutions for data types of different keys:

1) For keys of the non-binarycomparable type (refer to Appendix A), totalorderpartitioner uses binary distribution to find the index of the current K.

For example, the number of reducers is 5, and the four Partitions provided by partition file are [2, 4, 6, 8 ]. If the current key value pair is <4, "good"> Index = 1, index + 1 = 2, then the key value pair is sent to the second CER Cer. If a key value pair is <4.5, "good">, the binary search returns-3, and Adds 1 to-3 and returns the reducer to which the key value pair is going to go.

For some numeric data, the complexity of binary search is O (log (CER count), which is faster.

2) For keys of the binarycomparable type (or strings ). Strings can also be sorted alphabetically. In this way, we can also specify some divisions so that different string keys can be allocated to different reducers. The processing here is similar to the numerical type.

For example, if the number of reducers is 5, the partition file provides four partitions: ["ABC", "BCE", "EAA ", "FHC"] Then the "AB" string will be allocated to the first CER because it is smaller than the first division "ABC ".

However, unlike numeric data, string search and comparison cannot follow the numeric data comparison method. The string search method of the tire tree used by mapreducer. The time complexity O (m) of the search, M is the depth of the tree, and the space complexity O (255 m-1 ). It is a typical case of changing the space time.

Tire tree

Tire tree construction

Assume that the maximum depth of the tree is 3, which is divided into [AAad, AAAF, aaaeh, abbx]

Tairtree Structure

The tire tree in mapreduce mainly consists of two types of nodes:
1) innertirenode
Innertirenode is a long string containing 255 characters in mapreduce. The example in contains only 26 English letters.
2) leaf node {unslipttirenode, singesplittirenode, leaftirenode}
Unslipttirenode is a leaf node that does not contain the partition.
Singlesplittirenode is a leaf node that contains only one vertex division.
Leafnode is a leaf node that contains multiple points. (This is rare. This is the case only when the maximum depth of the tree is reached. It is rare in the actual operation process)

Tire tree search process

Example:
1) if the current key value pair is used, the leafnode in the figure will be found. In leafnode, the binary method is used to continue searching and return the index of AAD in the array. If no index is found, an index with the closest partition is returned.
2) If singlenode is found, if it is the same or small as singlenode, return its index. If it is larger than singlenode, return index + 1.
3) If nosplitnode is found, the previous index is returned. For example, the index of abbx In the partitioning array will be returned.

Totalorderpartitioner

The preceding section describes two requirements for partitioner: speed and load balancing. Tire tree improves the search speed, but how can we find such a partition file? Load Balancing can be achieved with all the divisions.

Inputsampler
Input sampling class, which can sample data in the input directory. Three sampling methods are provided.

Sample Structure

Sample Method Comparison table:

Class Name	Sampling Method	Constructor	Efficiency	Features
Splitsampler <K, V>	Sample the first N records	Total number of samples, number of samples	Highest
Randomsampler <K, V>	Traverse all data and perform random sampling	Sampling frequency, total number of samples, number of samples	Lowest
Intervalsampler <K, V>	Fixed interval sampling	Sampling frequency, number of partitions	Medium	Applicable to ordered data

The writepartitionfile method is critical. This method is to sort the samples provided by the sampling class first, and then write the samples (random method) and the CER number-1 to the partition file. In this way, the key value pair in each partition is approximately the same for the division generated by the sampled data, so that the load balancing function can be completed.

Totalorderpartitioner instance

?

12345678910111213141516171819202122232425262728293031323334353637383940

public class SortByTemperatureUsingTotalOrderPartitioner extends Configured        implements Tool{    @Override    public int run(String[] args) throws Exception    {        JobConf conf = JobBuilder.parseInputAndOutput(this, getConf(), args);        if (conf == null) {            return -1;        }        conf.setInputFormat(SequenceFileInputFormat.class);        conf.setOutputKeyClass(IntWritable.class);        conf.setOutputFormat(SequenceFileOutputFormat.class);        SequenceFileOutputFormat.setCompressOutput(conf, true);        SequenceFileOutputFormat                .setOutputCompressorClass(conf, GzipCodec.class);        SequenceFileOutputFormat.setOutputCompressionType(conf,                CompressionType.BLOCK);        conf.setPartitionerClass(TotalOrderPartitioner.class);        InputSampler.Sampler<IntWritable, Text> sampler = new InputSampler.RandomSampler<IntWritable, Text>(                0.1, 10000, 10);        Path input = FileInputFormat.getInputPaths(conf)[0];        input = input.makeQualified(input.getFileSystem(conf));        Path partitionFile = new Path(input, "_partitions");        TotalOrderPartitioner.setPartitionFile(conf, partitionFile);        InputSampler.writePartitionFile(conf, sampler);        // Add to DistributedCache        URI partitionUri = new URI(partitionFile.toString() + "#_partitions");        DistributedCache.addCacheFile(partitionUri, conf);        DistributedCache.createSymlink(conf);        JobClient.runJob(conf);        return 0;    }     public static void main(String[] args) throws Exception {        int exitCode = ToolRunner.run(                new SortByTemperatureUsingTotalOrderPartitioner(), args);        System.exit(exitCode);    }}

The sample program references to: http://www.cnblogs.com/funnydavid/archive/2010/11/24/1886974.html

Appendix
Text is of the binarycomparable and writeablecomparable type.
Booleanwritable, bytewritable, doublewritable, md5hash, intwritable, floatwritable, longwritable, and nullwriable are all writeablecomparable. For details, refer:

Appendix

• Tags: hadoop, mapreduce, partition

You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

 

Mapreduce-partition analysis (transfer)