"Hadoop Distributed Deployment Five: distribution, basic testing and monitoring of distributed deployments"

1. Format Hadoop

Execute command to/opt/app/hadoop-2.5.0 directory: Bin/hdfs Namenode-format

Execute the following (the successful format does not have nothing to format)

2. Start DFS

Execute command (under/opt/app/hadoop-2.5.0/directory): sbin/start-dfs.sh

After the execution of the effect is like (you can see 3 Datanode has been started, NameNode and Secondarynamenode have started up)

　　　　　　

And then, on each of these three machines, look at the boot.

Node One (hadoop-senior.zuoayn.com)

　　　　　　

Node two (hadoop-senior02.zuoyan.com)

　　　　　　

Node three (hadoop-senior03.zuoyan.com)

　　　　　　

You can see that the tasks performed on each node were designed by us at the time, and my situation is lucky that there are no errors that start up, but sometimes they appear

: Temporary failure in name resolutionop-senior02.zuoyan.com

: Temporary failure in name resolutionop-senior.zuoyan.com

This is due to the fact that the problem with copying a virtual machine occurs because the domain name resolution problem machine restarts, and there is another way to resolve

is to go to each machine and start using the command individually: sbin/hadoop-daemon.sh start Datanode

Open your browser to view the startup situation:

Enter the URL: http://hadoop-senior.zuoyan.com:50070 (This is the IP address of the first host, because I am mapped to the Windows hosts, so it can also be accessed through this host name)

Click on the Livenodes page to see the following interface

This interface shows our nodes.

　　　　　　　　　　　　

Use some commands to test

Create directory command: Bin/hdfs dfs-mkdir-p/user/beifeng/tmp/conf

　　　　　　　　

Upload file command: Bin/hdfs dfs-put/etc/hadoop/*.-site.xml/user/beifeng/tmp/conf

　　　　　　　　

Read File command: Bin/hdfs Dfs-text/user/beifeng/tmp/conf/core-site.xml (is the successful read out)

　　　　　　　　

　　　　　　　　

　　　　　　　　

3. Start yarn

(in/opt/app/hadoop-2.5.0 directory) use command: sbin/start-yarn.sh

　　　　

When I started yarn, my first problem was that ResourceManager didn't start up, either on the first node or on the second node, there was no ResourceManager.

Log information is as follows

　　　　　　

Finally found a solution in open source China

　　　　　　　　Namenode and Resourcemanger If they are not the same machine and cannot start yarn on the namenode, yarn should be started on the machine where the Resoucemanager is located.

4. Test the MapReduce program

First create a directory to hold the input data command: Bin/hdfs dfs-mkdir-p/user/beifeng/mapreduce/wordcount/input

　　　　　　　　

Upload file to File system command: Bin/hdfs dfs-put/opt/modules/hadoop-2.5.0/wc.input/user/beifeng/mapreduce/wordcount/input

　　　　　　　　　

Use the command to see if the file uploaded successfully command: Bin/hdfs dfs-ls/user/beifeng/mapreduce/wordcount/input (you can see that wc.input is already in this directory)

　　　　　　　　　　

　　　　　　　　

Start using yarn to run the WordCount program after you've finished your preparation

Command: Bin/yarn jar Share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar wordcount/user/beifeng/mapreduce/ Wordcount/input/user/beifeng/mapreduce/wordcount/output

The program has started running on the machine.

　　　　　　　　

　　　　　　　　

Effects that you see from a Web page

　　　　　　　　

　　　　　　　　

Finally, use the HDFS command to see the results of the WORDCOUNT statistics command: bin/hdfs-dfs-text/user/beifeng/mapreduce/wordcount/output/part*

　　　　　　　　

　　　　　　　　

To the end of this configuration, but the rest of the environment problem solving and cluster base testing

　　　　

　　　　　　　　

　　　　　　　　

　　　　　　　　

　　

　　　　　　

"Hadoop Distributed Deployment Five: distribution, basic testing and monitoring of distributed deployments"