Implementation and performance of Hadoop reference design: A preliminary test of Hadoop performance

Name Node/second name Node specification (total two servers):

Datanode/http://www.aliyun.com/zixun/aggregation/17034.html ">tasktracker Specification:

Cabinet Specification:

Hadoop Performance Preliminary test

Based on the Hadoop cluster set up above, the standard test components are used to validate the scenarios and performance tests are performed using the Hadoop performance Benchmarking Suite Hibench.

Nnbench

Test Objective: Load test the hardware and configuration of Namenode.

Parameter settings:

Maps = 12

Reduces = 6

Test command

In the Hadoop installation directory (typically/usr/lib/hadoop/), enter:

[Root@hadoop3 hadoop]# Hadoop jar Hadoop-test-1.0.3-intel.jar nnbench-operation create_write-maps 12-reduces 6- BlockSize 1-bytestowrite 0 numberoffiles 1000-r eplicationfactorperfile 3-readfileafteropen True

Test Pass criteria:

There are no errors during command execution.

Sort

Test objective: To test the performance of the entire MapReduce system.

Parameter settings:

Test command:

The first step is to generate random data in a directory:

Hadoop jar Hadoop-examples-1.0.3-intel.jar Randomwriter random-data

The second step calls the sort program:

Hadoop jar Hadoop-examples-1.0.3-intel.jar Sort Random-data sorted-data

The third step checks that the results of the second output are sorted correctly:

Hadoop jar Hadoop-test-1.0.3-intel.jar Testmapredsort-sortinput random-data-sortoutput sorted-data

Test Pass criteria:

Output after the third step:

success! Validated the MapReduce framework ' s ' sort ' successfully.

Testdfsio

Test Purpose: Test HDFs I/O performance. The system uses MapReduce to read and write files at the same time.

Parameter settings:

Nrfiles:10

filesize:1000

Test command:

Hadoop jar Hadoop-test-1.0.3-intel.jar Testdfsio-write-nrfiles 10-filesiz e 1000

After the test finishes, the following results are output:

-----Testdfsio-----: Write

Da te & Time:thu Ma y-03:01:56 CST 2013

Number o f files:10

Total MBytes processed:10000

THR Oughput mb/sec:92.87206872533086

Average IO Rate mb/sec:98.6314926147461

IO Rate STD deviation:23.668972646690577

Test Exec Time sec:47.991