MapReduce Go Heavy

One: Background

Many data sources in the data are a lot of duplication, we need to remove the duplicate data, which is also known as data cleansing, mapreduce from the map to the reduce side of the shuffle process is inherently deduplication function, but this is the output key as a reference to go heavy. So we can read the map end into value as the key output, it can be very convenient to implement the heavy.

Second: Technology realization

#需求 have two files File0 and file1. Merge the contents of the two files to heavy.

#file0的内容如下:

[Java]View PlainCopy

1. 1
2. 1
3. 2
4. 2
5. 3
6. 3
7. 4
8. 4
9. 5
10. 5
11. 6
12. 6
13. 7
14. 8
15. 9

The contents of File1 are as follows:

[Java]View PlainCopy

1. 1  
2. 9  
3. 9  
4. 8  
5. 8  
6. 7  
7. 7  
8. 6  
9. 6  
10. 5  
11. 5  
12. 4  
13. 4  
14. 2  
15. 1  
16. 2  

Code implementation:

[Java]View PlainCopy

2. Public class Distincttest {
4. //define Input path
6. private static final String Input_path = "hdfs://liaozhongmin:9000/distinct_file/*";
8. //define Output path
10. private static final String Out_path = "Hdfs://liaozhongmin:9000/out";
14. public static void Main (string[] args) {
18. try {
20. //Create configuration information
22. Configuration conf = new configuration ();
28. //Create File System
30. FileSystem FileSystem = Filesystem.get (new URI (Out_path), conf);
32. //If the output directory exists, we will delete
34. if (filesystem.exists (new Path (Out_path))) {
36. Filesystem.delete (new Path (Out_path), true);
38. }
42. //Create a task
44. Job Job = New Job (conf, distincttest.   Class.getname ());
48. //1.1 Set Input directory and set the input data format class
50. Fileinputformat.setinputpaths (Job, Input_path);
52. Job.setinputformatclass (Textinputformat.   Class);
56. //1.2 Setting the Custom mapper class and setting the type of key and value of the map function output data
58. Job.setmapperclass (distinctmapper.   Class);
60. Job.setmapoutputkeyclass (Text.   Class);
62. Job.setmapoutputvalueclass (Text.   Class);
66. //1.3 sets the number of partitions and reduce (the number of reduce, which corresponds to the number of partitions, because the partition is one, so the number of reduce is also one)
68. Job.setpartitionerclass (Hashpartitioner.   Class);
70. Job.setnumreducetasks (1);
74. //1.4 Sort
76. //1.5
78. Job.setcombinerclass (distinctreducer.   Class);
80. //2.1 Shuffle copies data from the map side to the reduce side.
82. //2.2 Specifies the type of reducer class and output key and value
84. Job.setreducerclass (distinctreducer.   Class);
86. Job.setoutputkeyclass (Text.   Class);
88. Job.setoutputvalueclass (Text.   Class);
92. //2.3 Specifies the path of the output and the format class for setting the output
94. Fileoutputformat.setoutputpath (Job, new Path (Out_path));
96. Job.setoutputformatclass (Textoutputformat.   Class);
102. //Submit Job Exit
104. System.exit (Job.waitforcompletion (true)?  0: 1);
108. } catch (Exception e) {
110. E.printstacktrace ();
112. }
114. }
118. public static class Distinctmapper extends Mapper<longwritable, text, text, text>{
120. //define the key and value to write out
122. private Text Outkey = new text ();
124. private Text Outvalue = new text ("");
126. @Override
128. protected void map (longwritable key, text value, mapper<longwritable, text, text, Text>. Context context) throws IOException, interruptedexception {
130. //input key as value output (because)
132. Outkey = value;
136. //write the results out
138. Context.write (Outkey, Outvalue);
140. }
142. }
146. public static class Distinctreducer extends Reducer<text, text, text, text>{
148. @Override
150. protected Void reduce (text key, iterable<text> value, Reducer<text, text, text, Text>. Context context) throws IOException, interruptedexception {
152. //write out the key directly
154. Context.write (Key, new Text (""));
156. }
158. }
160. }

The results of the program run:

MapReduce Go Heavy