Hadoop cluster configuration

I have been checking on the Internet for a long time and have failed due to various details and errors. Today I am able to do it. The process is as follows:

 

Basic Environment:

Master cloud003 IP: 192.168.140.203

Slave cloud004IP: 192.168.140.204

Note: The Virtual Machine IP address must be configured in NAT mode,

Operating System: ubuntu

Now the installation is started. The process of downloading JAVA is skipped.

 

Install JAVA, HADOOP, and SSH on each machine. Be sure to install them in the same directory; install JAVA and Hadoop in the/usr/local directory and set the environment variables to the/etc/profile file, in this way, other users can access export JAVA_HOME =/usr/local/jdk1.6.0 _ 25 export CLASSPATH =.: $ CLASSPATH: $ JAVA_HOME/lib/dt. jar: $ JAVA_HOME/lib/tools. jarexport PATH = $ PATH: $ JAVA_HOME/binexport JRE_HOME = $ JAVA_HOME/jreexport HADOOP_HOME =/usr/local/hadoop-0.20.2export PATH = $ HADOOP_HOME/bin: $ PATHSSH log on to ssh-keygen-t dsa-p'-f ~ /. Ssh/id_dsa cat ~ /. Ssh/id_dsa.pub> ~ /. Ssh/authorized_keyschmod 644 authorized_keys is critical. You must ensure that authorized_keys only has read and write permissions on its owner, while others do not allow write permissions. Otherwise, SSH will not work. I have been depressed for a long time When configuring SSH.

 

The above operations are the same for both machines, and the following differences exist:

Copy id_dsa.pub on cloud003 to cloud004, and append the content to authorized_keys, similarly, the id_dsa.pub generated on cloud004 must be appended to the authorized_keys on cloud003 to test whether SSH is connected to ssh 192.168.140.203 on cloud003 or ssh 192.168.140.204 on cloud004. The following are the key points:Modify the Master file configuration in the conf folder in the Hadoop configuration file: the master file content is: cloud003

 

Slaves file content: cloud004

 

Conf/core-site.xml file:

<Configuration>

<Property> <name> fs. default. name </name> <value> hdfs: // cloud003: 9000 </value> </property> </configuration>

 

Conf/hdfs-site.xml file:

<Configuration>

<Property> <name> dfs. replication </name> <value> 2 </value> </property> </configuration>

 

Conf/mapred-site.xml file:

 

<Configuration> <property> <name> mapred. job. tracker </name> <value> cloud003: 9001 </value> </property> </configuration> set on Slave (the hdfs-site.xml does not need to be set) The contents of the Masters file: cloud003slaves file content is: cloud004conf/core-site.xml file content: <configuration> <property> <name> fs. default. name </name> <value> hdfs: // cloud003: 9000 </value> </property> </configuration> conf/mapred-site.xml file content: <configuration> <property> <name> mapred. job. tracker </name> <value> cloud003: 9001 </value> </property> </configuration>

Perform operations on cloud003 after configuration is complete.

1: bin/hadoop namenode-format # format

2:./start-dfs.sh

3:./start-mapred.sh

Test bin/hadoop jar./hadoop-0.20.2-examples.jar wordcount input/usr/root/output -- View Details bin/hadoop dfsadmin-report

The result is as follows:

 

Configured Capacity: 83492536320 (77.76 GB) Present Capacity: 72159903744 (67.2 GB) DFS Remaining: 72159100928 (67.2 GB) DFS Used: 802816 (784 KB) DFS Used %: 0% Under replicated blocks: 7 Blocks with primary upt replicas: 0 Missing blocks: 0 primary Datanodes available: 2 (2 total, 0 dead) Name: 192.168.140.203: 50010 Decommission Status: NormalConfigured Capacity: 41746268160 (38.88 GB) DFS Used: 401408 (392 KB) Non DFS Used: 5471211520 (5.1 GB) DFS Remaining: 36274655232 (33.78 GB) DFS Used %: 0% DFS Remaining %: 86.89% Last contact: Sat Jun 25 01:20:39 PDT 2011 Name: 192.168.140.204: 50010 Decommission Status: NormalConfigured Capacity: 41746268160 (38.88 GB) DFS Used: 401408 (392 KB) Non DFS Used: 5861421056 (5.46 GB) DFS Remaining: 35884445696 (33.42 GB) DFS Used %: 0% DFS Remaining %: 85.96% Last contact: Sat Jun 25 01:20:15 PDT 2011