Ubuntu under hadoop2.4 build cluster (standalone mode)

I. New users and user groups

Note: (This step in fact can not be used.) Just use a different user alone better)

1. Create a new user group

sudo addgroup Hadoop

2. Create a new user

sudo adduser-ingroup Hadoop Hadoop

3. Join Hadoop User Rights

sudo gedit  /etc/sudoers

Join a Hadoop user after opening the Sudoer file

watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvemhvdxpob3vqawfucxvhbg==/font/5a6l5l2t/fontsize/400/fill/ I0jbqkfcma==/dissolve/70/gravity/center ">

# User Privilege specificationroot    all= (all:all) allhadoop  all= (all:all) all

4. Log in with a Hadoop user

Second, install SSH

sudo apt-get install Openssh-server

watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvemhvdxpob3vqawfucxvhbg==/font/5a6l5l2t/fontsize/400/fill/ I0jbqkfcma==/dissolve/70/gravity/southeast ">

After the installation is complete. Start the service

Sudo/etc/init.d/ssh start

To see if the service started correctly: PS-E | grep ssh

Cluster, single-node mode is required to SSH without password login, first set up SSH without password landing this machine.

Input command

SSH  localhost

Enter Yes for first login

Set up a password login to generate the private key and public key

Ssh-keygen-t rsa-p ""

watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvemhvdxpob3vqawfucxvhbg==/font/5a6l5l2t/fontsize/400/fill/ I0jbqkfcma==/dissolve/70/gravity/southeast ">

Here we append the public key to Authorized_keys, which the user holds all the public key content that agrees to log on to the Sshclient user as the current user.

Cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

You can then use SSH Localhsot to log in without password.

Use Exit to sign out

watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvemhvdxpob3vqawfucxvhbg==/font/5a6l5l2t/fontsize/400/fill/ I0jbqkfcma==/dissolve/70/gravity/southeast ">

Third, install the Java environment

The previous tutorials are recommended to install Oracle's JDK, and OPENJDK is not recommended. Just according to Http://wiki.apache.org/hadoop/HadoopJavaVersions said. The new version number under OPENJDK 1.7 is no problem. Install openjdk 7 by command.

Keng

Sudoapt-getinstall OPENJDK-7-JREOPENJDK-7-JDK

View the installation results. Input command: Java-version. Results such as the following indicate a successful installation.

watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvemhvdxpob3vqawfucxvhbg==/font/5a6l5l2t/fontsize/400/fill/ I0jbqkfcma==/dissolve/70/gravity/southeast ">

To view the installation results, enter the command: Java-version, results such as the following indicate a successful installation.

watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvemhvdxpob3vqawfucxvhbg==/font/5a6l5l2t/fontsize/400/fill/ I0jbqkfcma==/dissolve/70/gravity/southeast ">

Iv. installation of Hadoop 2.4.1

2.4.1: http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.4.1/hadoop-2.4.1.tar.gz, installation tutorial mainly participated in the official tutorial/HTTP// Hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/singlecluster.html.

After downloading, unzip to/usr/local/.

Then change the directory name to Hadoop

Give the user read and write permission to the directory (this problem is very pit, I was configured at the time by some method pits, do not understand the file permissions of things)

Some suggestions are:

sudo chmod 774/usr/local/hadoop

But after I use this command. The directory is all hidden. Can not open. Finally, I deleted the Hadoop directory. Use the following to solve the problem.

sudo  chown-r hadoop:hadoop   /usr/local/hadoop

  
Configure ~/.BASHRC

Before you configure This file, you need to know the Java installation path. Use to set the JAVA_HOME environment variable to view the installation path using the following command line

Update-alternatives--config Java

Running results such as the following:

watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvemhvdxpob3vqawfucxvhbg==/font/5a6l5l2t/fontsize/400/fill/ I0jbqkfcma==/dissolve/70/gravity/southeast ">

Configure the. bashrc file

sudo gedit ~/.BASHRC

#HADOOP VARIABLES startexport Java_home=/usr/lib/jvm/java-7-openjdk-i386export hadoop_install=/usr/local/ Hadoopexport path= $PATH: $HADOOP _install/binexport path= $PATH: $HADOOP _install/sbinexport hadoop_mapred_home=$ Hadoop_installexport hadoop_common_home= $HADOOP _installexport hadoop_hdfs_home= $HADOOP _installexport YARN_HOME=$ Hadoop_installexport hadoop_common_lib_native_dir= $HADOOP _install/lib/nativeexport hadoop_opts= "- Djava.library.path= $HADOOP _install/lib "#HADOOP VARIABLES END

Run the following command. To make the joined environment variable effective:

SOURCE ~/.BASHRC

Edit/usr/local/hadoop/etc/hadoop/hadoop-env.sh

Run the following command . Open the Edit form for the file

sudo gedit/usr/local/hadoop/etc/hadoop/hadoop-env.sh

Locate the Java_home variable and change this variable such as the following

Export java_home==/usr/lib/jvm/java-7-openjdk-i386   

watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvemhvdxpob3vqawfucxvhbg==/font/5a6l5l2t/fontsize/400/fill/ I0jbqkfcma==/dissolve/70/gravity/southeast ">

Five, Test WordCount

Stand-alone mode is installed. The following is verified if the installation was successful by running the Hadoop self-wordcount instance

Create input directory under/usr/local/hadoop path

sudo mkdir input

Copy README.txt to input

CP README.txt Input

Run WordCount

Bin/hadoop Jar Share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.4.0-sources.jar Org.apache.hadoop.examples.WordCount Input Output

watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvemhvdxpob3vqawfucxvhbg==/font/5a6l5l2t/fontsize/400/fill/ I0jbqkfcma==/dissolve/70/gravity/southeast ">

Perform such as the following

Run Cat output/*. View character Statistics results

Results such as the following

watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvemhvdxpob3vqawfucxvhbg==/font/5a6l5l2t/fontsize/400/fill/ I0jbqkfcma==/dissolve/70/gravity/southeast ">

Ubuntu under hadoop2.4 build cluster (standalone mode)