Hadoop configuration rack awareness

Zhou haihan? 2013.7.24abloz.com if the device connection level is divided into three layers, the first layer switch d1 is connected to multiple switches rk1, rk2, rk3, rk4. Each switch corresponds to a rack. D1 (rk1 (hs11, hs12,), rk2 (hs21, hs22,), rk3 (hs31, hs32,), rk4 (hs41, hs42 ,),) you can use a program or script to map the host to the device.

Zhou haihan? 2013.7.24 http://abloz.com if the device link layer is divided into three layers, the first layer switch d1 connected to multiple switches rk1, rk2, rk3, rk4,. Each switch corresponds to a rack. D1 (rk1 (hs11, hs12,), rk2 (hs21, hs22,), rk3 (hs31, hs32,), rk4 (hs41, hs42 ,),) you can use a program or script to map the host to the device.

Zhou haihan? 2013.7.24

Http://abloz.com

If the device connection layer is divided into three layers, the first layer switches d1 are connected to multiple switches rk1, rk2, rk3, rk4 ,.... Each vswitch corresponds to a rack.

D1 (rk1 (hs11, hs12 ,...), Rk2 (hs21, hs22 ,...), Rk3 (hs31, hs32 ,...), Rk4 (hs41, hs42 ,...),...)

You can use a program or script to map the host to the device. For example, use python to generate a topology. py:

Then configure

Topology. script. file. name
/Home/hadoop/hadoop-1.1.2/conf/topology. py
The script name that shoshould be invoked to resolve DNS names
NetworkTopology names. Example: the script wocould take host. foo. bar as
Argument, and return/rack1 as the output.

Python rack script:

[Hadoop @ hs11 conf] $ cat topology. py
#! /Usr/bin/env python

"'
This script used by hadoop to determine network/rack topology. It
Shocould be specified in hadoop-site.xml via topology. script. file. name
Property.
Topology. script. file. name
/Home/hadoop/hadoop-1.1.2/conf/topology. py

To generate dict:
For I in range (xx ):
# Print "\" hs % d \ ": \"/rk % d/hs % d \ "," % (I, (I-1)/10, I)

Print "\" hs % d \ ": \"/rk % d \ "," % (I, (I-1)/10)

Andy 2013.7.23
"'

Import sys
From string import join

DEFAULT_RACK = '/rk0 ′;

RACK_MAP = {
"Hs11": "/rk1 ″,
"Hs12": "/rk1 ″,
"Hs13": "/rk1 ″,
"Hs14": "/rk1 ″,
"Hs15": "/rk1 ″,
"Hs16": "/rk1 ″,
"Hs17": "/rk1 ″,
"Hs18": "/rk1 ″,
"Hs19": "/rk1 ″,
"Hs20": "/rk1 ″,
"Hs21": "/rk2 ″,
"Hs22": "/rk2 ″,
"Hs23": "/rk2 ″,
"Hs24": "/rk2 ″,
"Hs25": "/rk2 ″,
"Hs26": "/rk2 ″,
"Hs27": "/rk2 ″,
"Hs28": "/rk2 ″,
"Hs29": "/rk2 ″,
"Hs30": "/rk2 ″,
"Hs31": "/rk3 ″,
"Hs32": "/rk3 ″,
"Hs33": "/rk3 ″,
"Hs34": "/rk3 ″,
"Hs35": "/rk3 ″,
"Hs36": "/rk3 ″,
"Hs37": "/rk3 ″,
"Hs38": "/rk3 ″,
"Hs39": "/rk3 ″,
"Hs40": "/rk3 ″,
"Hs41": "/rk4 ″,
"Hs42": "/rk4 ″,
"Hs43": "/rk4 ″,
"Hs44": "/rk4 ″,
"Hs45": "/rk4 ″,
"Hs46": "/rk4 ″,

...

"10.10.20.11": "/rk1 ″,
"10.10.20.12": "/rk1 ″,
"10.10.20.13": "/rk1 ″,
"10.10.20.14": "/rk1 ″,
"10.10.20.15": "/rk1 ″,
"10.10.20.16": "/rk1 ″,
"10.10.20.17": "/rk1 ″,
"10.10.20.18": "/rk1 ″,
"10.10.20.19": "/rk1 ″,
"10.10.20.20": "/rk1 ″,
"10.10.20.21": "/rk2 ″,
"10.10.20.22": "/rk2 ″,
"10.10.20.23": "/rk2 ″,
"10.10.20.24": "/rk2 ″,
"10.10.000025": "/rk2 ″,
"10.10.20.26": "/rk2 ″,
"10.10.20.27": "/rk2 ″,
"10.10.20.28": "/rk2 ″,
"10.10.20.29": "/rk2 ″,
"10.10.20.30": "/rk2 ″,
"10.10.20.31": "/rk3 ″,
"10.10.20.32": "/rk3 ″,
"10.10.20.33": "/rk3 ″,
"10.10.000034": "/rk3 ″,
"10.10.20.35": "/rk3 ″,
"10.10.20.36": "/rk3 ″,
"10.10.20.37": "/rk3 ″,
"10.10.20.38": "/rk3 ″,
"10.10.20.39": "/rk3 ″,
"10.10.20.40": "/rk3 ″,
"10.10.20.41": "/rk4 ″,
"10.10.20.42": "/rk4 ″,
"10.10.20.43": "/rk4 ″,
"10.10.000044": "/rk4 ″,
"10.10.20.45": "/rk4 ″,
"10.10.000046": "/rk4 ″,

...
}

If len (sys. argv) = 1:
Print DEFAULT_RACK
Else:
Print join ([RACK_MAP.get (I, DEFAULT_RACK) for I in sys. argv [1:], "")

In the past, I returned

"Hs11": "/rk1/hs11 ″,

The following error is reported when the mapreduce program is executed:

Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201307241502_0003, Tracking URL = http: // hs11: 50030/jobdetails. jsp? Jobid = job_201307241502_0003
Kill Command =/home/hadoop/hadoop-1.1.2/libexec/../bin/hadoop job? -Kill job_201307241502_0003
Hadoop job information for Stage-1: number of mappers: 0; number of worker CERs: 0
18:38:11, 854 Stage-1 map = 100% ,? Reduces = 100%
Ended Job = job_201307241502_0003 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http: // hs11: 50030/jobdetails. jsp? Jobid = job_201307241502_0003
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exe c. MapRedTask
MapReduce Jobs Launched:
Job 0 :? HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

Http: // hs11: 50030/jobdetails. jsp? Jobid = job_201307241502_0002? You can see:

Job initialization failed:

Java. lang. NullPointerException

At? Org. apache. hadoop. mapred. JobTracker. resolveAndAddToTopology (JobTracker. java: 2751)
At? Org. apache. hadoop. mapred. JobInProgress. createCache (JobInProgress. java: 578)
At? Org. apache. hadoop. mapred. JobInProgress. initTasks (JobInProgress. java: 750)

At org. apache. hadoop. mapred. JobTracker. initJob (JobTracker. java: 3775)

At? Org. apache. hadoop. mapred. EagerTaskInitializationListener $ InitJob. run (EagerTaskInitializationListener. java: 90)
At? Java. util. concurrent. ThreadPoolExecutor $ Worker. runTask (ThreadPoolExecutor. java: 886)
At? Java. util. concurrent. ThreadPoolExecutor $ Worker. run (ThreadPoolExecutor. java: 908)
At java. lang. Thread. run (Thread. java: 662)

The system does not need to return the ns or hostname of the device in the script when configuring rack sensitivity. The system will automatically add the device. Change topology. py to the preceding one, and the system runs correctly.

Related blog posts:

1. Hadoop prints configuration variables
2. ClassNotFoundException in hadoop
3. Hadoop ubuntu cluster Installation

Original article address: hadoop configuration rack awareness. Thanks to the original author for sharing.