MapReduce Performance Analysis Experiment

A number of experimental tests have been conducted on MapReduce recently, as required by the project, as documented below.

Test environment

3 VM VMS, all Ubuntu systems, 1G memory, Hadoop 2.6.0

1 sets of NameNode (Master)

3 sets of DataNode (Slave)

where Master and 2 slave (Slave2, Slave3) are located in a strong physical machine, another 1Slave (SLAVE1) is located in a poorly configured physical machine.

Data preparation

A total of 28 text files, each file about 12M, a total of about 330M of data

The content is roughly

Distribution of tasks in experiment 1 nodes

Our test program is the basic word counting program.

 PackageMyPackage;Importjava.io.IOException;ImportJava.util.StringTokenizer;Importorg.apache.hadoop.conf.Configuration;ImportOrg.apache.hadoop.fs.Path;Importorg.apache.hadoop.io.IntWritable;ImportOrg.apache.hadoop.io.Text;ImportOrg.apache.hadoop.mapreduce.Job;ImportOrg.apache.hadoop.mapreduce.Mapper;ImportOrg.apache.hadoop.mapreduce.Reducer;ImportOrg.apache.hadoop.mapreduce.lib.input.FileInputFormat;ImportOrg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;ImportOrg.apache.hadoop.util.GenericOptionsParser;  Public classWordCount { Public Static classMapextendsMapper<object, text, text, intwritable> {    Private Final StaticIntwritable one =NewIntwritable (1); PrivateText Word =NewText ();  Public voidMap (Object key, Text value, context context)throwsIOException, interruptedexception {String line=value.tostring (); StringTokenizer Tokenizer=NewStringTokenizer (line);  while(Tokenizer.hasmoretokens ()) {Word.set (Tokenizer.nexttoken ());        Context.write (Word, one); }    } }    Public Static classReduceextendsReducer<text, Intwritable, Text, intwritable> {     Public voidReduce (Text key, iterable<intwritable>values, context context)throwsIOException, interruptedexception {intsum = 0;  for(intwritable val:values) {sum+=Val.get (); } context.write (Key,Newintwritable (sum)); } }    Public Static voidMain (string[] args)throwsException {Configuration conf=NewConfiguration (); Job Job=NewJob (conf, "WordCount"); Job.setjarbyclass (WordCount.class); Job.setoutputkeyclass (Text.class); Job.setoutputvalueclass (intwritable.class); Job.setcombinerclass (Reduce.class); Job.setmapperclass (Map.class); Job.setreducerclass (Reduce.class); Fileinputformat.addinputpath (Job,NewPath (args[0])); Fileoutputformat.setoutputpath (Job,NewPath (args[1])); Job.waitforcompletion (true); }        }

Experimental results

It took about 11 minutes to complete the mission.

A total of 28 map tasks were launched

28 map tasks are not evenly distributed on 3 slave.

attempt	state	status	node	logs	start Time	finish time	elapsed time	note
Attempt_1430534608975_0001_m_000000_0	Succeeded	Map	/default-rack/slave3:8042	Logs	Sat, 02:44:02 GMT	Sat, 02:47:03 GMT	3mins, 0sec
Attempt_1430534608975_0001_m_000001_0	Succeeded	Map	/default-rack/slave3:8042	Logs	Sat, 02:44:03 GMT	Sat, 02:47:08 GMT	3mins, 5sec
Attempt_1430534608975_0001_m_000002_0	Succeeded	Map	/default-rack/slave3:8042	Logs	Sat, 02:44:03 GMT	Sat, 02:47:08 GMT	3mins, 5sec
Attempt_1430534608975_0001_m_000003_0	Succeeded	Map	/default-rack/slave3:8042	Logs	Sat, 02:44:03 GMT	Sat, 02:47:07 GMT	3mins, 4sec
Attempt_1430534608975_0001_m_000004_0	Succeeded	Map	/default-rack/slave3:8042	Logs	Sat, 02:44:03 GMT	Sat, 02:47:08 GMT	3mins, 5sec
Attempt_1430534608975_0001_m_000005_0	Succeeded	Map	/default-rack/slave3:8042	Logs	Sat, 02:44:03 GMT	Sat, 02:47:12 GMT	3mins, 9sec
Attempt_1430534608975_0001_m_000006_0	Succeeded	Map	/default-rack/slave3:8042	Logs	Sat, 02:44:03 GMT	Sat, 02:47:08 GMT	3mins, 5sec
Attempt_1430534608975_0001_m_000007_0	Succeeded	Map	/default-rack/slave3:8042	Logs	Sat, 02:44:03 GMT	Sat, 02:47:00 GMT	2mins, 57sec
Attempt_1430534608975_0001_m_000008_0	Succeeded	Map	/default-rack/slave2:8042	Logs	Sat, 02:44:02 GMT	Sat, 02:45:31 GMT	1mins, 28sec
Attempt_1430534608975_0001_m_000009_0	Succeeded	Map	/default-rack/slave2:8042	Logs	Sat, 02:44:02 GMT	Sat, 02:45:55 GMT	1mins, 52sec
Attempt_1430534608975_0001_m_000010_0	Succeeded	Map	/default-rack/slave2:8042	Logs	Sat, 02:44:02 GMT	Sat, 02:45:59 GMT	1mins, 57sec
Attempt_1430534608975_0001_m_000011_0	Succeeded	Map	/default-rack/slave2:8042	Logs	Sat, 02:44:02 GMT	Sat, 02:45:54 GMT	1mins, 52sec
Attempt_1430534608975_0001_m_000012_0	Succeeded	Map	/default-rack/slave2:8042	Logs	Sat, 02:44:02 GMT	Sat, 02:45:31 GMT	1mins, 28sec
Attempt_1430534608975_0001_m_000013_0	Succeeded	Map	/default-rack/slave2:8042	Logs	Sat, 02:44:02 GMT	Sat, 02:45:55 GMT	1mins, 52sec
Attempt_1430534608975_0001_m_000014_1	Succeeded	Map	/default-rack/slave2:8042	Logs	Sat, 02:47:18 GMT	Sat, 02:47:23 GMT	4sec
Attempt_1430534608975_0001_m_000015_0	Succeeded	Map	/default-rack/slave1:8042	Logs	Sat, 02:45:08 GMT	Sat, 02:50:42 GMT	5mins, 33sec
Attempt_1430534608975_0001_m_000016_0	Succeeded	Map	/default-rack/slave1:8042	Logs	Sat, 02:45:08 GMT	Sat, 02:50:57 GMT	5mins, 48sec
Attempt_1430534608975_0001_m_000017_1	Succeeded	Map	/default-rack/slave2:8042	Logs	Sat, 02:47:48 GMT	Sat, 02:47:52 GMT	4sec
Attempt_1430534608975_0001_m_000018_1	Succeeded	Map	/default-rack/slave2:8042	Logs	Sat, 02:48:18 GMT	Sat, 02:48:23 GMT	4sec
Attempt_1430534608975_0001_m_000019_0	Succeeded	Map	/default-rack/slave1:8042	Logs	Sat, 02:45:08 GMT	Sat, 02:50:39 GMT	5mins, 30sec
Attempt_1430534608975_0001_m_000020_1	Succeeded	Map	/default-rack/slave3:8042	Logs	Sat, 02:48:48 GMT	Sat, 02:48:53 GMT	4sec
Attempt_1430534608975_0001_m_000021_1	Succeeded	Map	/default-rack/slave3:8042	Logs	Sat, 02:48:03 GMT	Sat, 02:48:09 GMT	6sec
Attempt_1430534608975_0001_m_000022_0	Succeeded	Map	/default-rack/slave2:8042	Logs	Sat, 02:45:38 GMT	Sat, 02:46:53 GMT	1mins, 14sec
Attempt_1430534608975_0001_m_000023_0	Succeeded	Map	/default-rack/slave2:8042	Logs	Sat, 02:46:02 GMT	Sat, 02:47:08 GMT	1mins, 6sec
Attempt_1430534608975_0001_m_000024_0	Succeeded	Map	/default-rack/slave2:8042	Logs	Sat, 02:46:01 GMT	Sat, 02:47:00 GMT	58sec
Attempt_1430534608975_0001_m_000025_0	Succeeded	Map	/default-rack/slave2:8042	Logs	Sat, 02:46:01 GMT	Sat, 02:46:53 GMT	51sec
Attempt_1430534608975_0001_m_000026_0	Succeeded	Map	/default-rack/slave2:8042	Logs	Sat, 02:46:01 GMT	Sat, 02:47:08 GMT	1mins, 6sec
Attempt_1430534608975_0001_m_000027_0	Succeeded	Map	/default-rack/slave2:8042	Logs	Sat, 02:46:55 GMT	Sat, 02:47:15 GMT	19sec

Hadoop is still quite intelligent, she has speculative execution mechanism, can proactively discover some of the slower performance of the node, assigned to them less tasks, and to better performance of the nodes assigned more tasks.

As you can see, in our experiment, the distribution situation is: Slave1-3, slave2-15, slave3-10. SLAVE1 performance is poor and the results are assigned to the fewest tasks.

The role of Experiment 2 combiner

You may have noticed that, in contrast to the standard official WordCount program, our code adds a line

Job.setcombinerclass (Reduce.  Class);

What does she do with it?

Here is the shuffle process of MapReduce: In a clustered environment, the map task and the reduce task are not on the same node, and the reduce task needs to pull the result of the map task across nodes to other nodes, we can reduce the map task as much as possible. The amount of output data, so as to achieve the purpose of improving overall performance.

The combine here is to merge the intermediate results of the map task, reducing the amount of output data from the map task, and reducing the amount of data that overflows to disk, thereby improving performance.

Let's take this line out of the code to see how efficient the task is.

The result ran for 20 minutes, stopped not to run, failed to complete ....  Suspicion was another physical machine that was unable to resist, causing the whole task to snap. Now turn off Slave1, only enable Slave2 and Slave4 for Datanode, continue to test.

It only took about 5 minutes to open the combiner.

Map Task Distribution:

Slave2-16, slave3-12

Close Combiner,

The result is still running stuck ... The explanation is not to run the slow node caused but in this case does not enable combiner or even does not work ... Later, in the case of not enabling combiner only in Slave2 and Slave3 as a datanode, and then conducted several tests, almost every time ran stuck ...

Here at least one point: in the case of large data volume, this test is about 300M, do not enable combiner, will cause a heavy load on the network, and even cause the entire task can not be completed.

Combiner in the end is what, recommended to see this article explains in great detail the shuffle process. http://liouwei20051000285.blog.163.com/blog/static/252367420116208743834/

Simply put, the map phase shuffle process:

Partition: Decide Key/value should have a reducer to deal with, put this information in the buffer

Combiner: For projects of the same key, we call this combiner. Greatly reduces the amount of data that overflows to disk and the amount of data transmitted over the network.

Merge: Merges multiple overflow files on the disk into an overflow file. This process, if you have the same key will do combine.

Reduce phase Shuffle process:

Copy: Pull the data and copy the completed map task to the memory area of the Reduce task node

Merge: Similar to the map phase, generates numerous overflow files on disk

Input: Continuous merge to form the final reducer input file

MapReduce Performance Analysis Experiment