Build a local hadoop test environment

Document directory

• Namenode node configuration
• Hdfs test environment Configuration
• Job-tracker Configuration

Operating System: Ubuntu 9.10 download hadoop: hadoop-0.20.1.tar.gz install dependency software:

Java 1.6x or later

$ Sudo apt-get install sun-java6-bin

The conf/hadoo-env.sh is located in the JAVA_HOME variable:

JAVA_HOME =/usr/lib/jvm/java-6-sun

Ssh and sshd

$ Sudo apt-get install openssh-serverhadoop test environment configuration (single-host, simulated distributed environment)
Namenode node configuration

Conf/core-site.xml:

<Configuration>
<Property>
<Name> fs. default. name </name>
<Value> hdfs :/// localhost: 9000 </value>
</Property>
</Configuration> hdfs test environment configuration

Conf/hdfs-site.xml: no redundancy required for data blocks

<Configuration>
<Property>
<Name> dfs. replication </name>
<Value> 1 </value>
</Property>
</Configuration> job-tracker configuration

Conf/mapred-site.xml:

<Configuration>
<Property>
<Name> mapred. job. tracker </name>
<Value> localhost: 9001 </value>
</Property>
</Configuration> password-free ssh login configuration

 

$ Ssh localhost
# If not, perform the following Configuration:
$ Ssh-keygen-t dsa-p'-f ~ /. Ssh/id_dsa
$ Cat ~ /. Ssh/id_dsa.pub> ~ /. Ssh/authorized_keys

 

Initialize HDFS and start hadoop

 

$ Bin/hadoop namenode-format
$ Bin/start-all.sh
# All running logs are in the $ {HADOOP_LOG_DIR} directory (default:
$ {HADOOP_HOME}/logs). Check NameNode and JobTracker now:

NameNode-
Http: // localhost: 50070/

JobTracker-
Http: // localhost: 50030/

Official documentation: Hadoop Quick Start