The problems and solutions in the process and construction of Hadoop environment

See most of the online Hadoop set up a tutorial, which part of the construction process can not be successful, now set up the Hadoop environment to build the basic steps and build the process encountered problems and corresponding solutions

the tools used are as follows:

linux:centos6.5

jdk:jdk-8u101-linux-x64.tar.gz corresponding official website download address is as follows: http://eclipse.org/downloads/

Virtual machine: VMware Workstation Pro

Hadoop:hadoop-0.20.2-tar.gz

Step One: Configure the SSH service

First, CentOS6.5 with the SSH service, using the command PS-E | grep ssh, if the service is already installed, the word sshd will appear, and if the service is not installed, you can install the service online using yum install SSH.

And then

Execute the following command: Cd/root

Execute the following command: ssh-keygen-t RSA

Then always enter, after the configuration succeeds, execute LS, because the directory is a hidden directory, so direct CD. SSH, and then LS, will find the following two files id_rsa.pub Authorized_keys, this SSH service configuration is complete

Step Two: Install JDK8

①. Create a Java directory under/usr, and then copy jdk-8u101-linux-x64.tar.gz to that directory.

Execute the following command: CD/USR

mkdir Java

Cp/home/**/desktop/jdk-8u101-linux-x64.tar.gz/usr/java

TAR-ZXVF jdk-8u101-linux-x64.tar.gz

②. Configure the following in/etc/profile

Execute the following command: Vi/etc/profile

Configuration content is as follows:

Export java_home=/usr/java/jdk1.6.0_22
Export classpath=.: $JAVA _home/jre/lib/rt.jar: $JAVA _home/lib/dt.jar: $JAVA _home/lib/tools.jar
Export path= $PATH: $JAVA _home/bin
So we set up the JDK, under the CentOS Source/etc/profile can be effective

Step Three: Install Hadoop

①. Create a new directory/usr/hadoop, copy hadoop-0.20.2-tar.gz to the directory, modify/etc/profile file, and add it at the end of the file

The following content:

Export hadoop_home=/usr/hadoop/hadoop-0.20.2
Export path= $HADOOP _home/bin: $PATH

②.CD to/usr/hadoop/hadoop-0.20.2/conf to configure the following four files,

respectively: Hadoop-env.shcore-site.xml hdfs-site.xml mapred-site.xml

1> Configure the following in hadoop-env.sh

Export java_home=/usr/java/jdk1.8.0_101

Export hadoop_classpath=/usr/hadoop/hadoop-0.20.2

2> Configure the following in Core-site.xml

<property>

<name>fs.default.name</name>

<value>hdfs://localhost:9000</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/usr/hadoop-0.20.2</value>

</property>

3> Configure the following in Hdfs-site.xml

<property>

<name>dfs.data.dir</name>

<value>/usr/hadoop-0.20.2/data</value>

</property>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

4> Configure the following in Mapred-site.xml

<property>

<name>mapred.job.tracker</name>

<value>localhost:9001</value>

</property>

<property>

<name>mapred.chlid.tmp</name>

<value>/opt/temp</value>

</property>

One last step

①. Shut down the firewall, execute command under root user/etc/init.d/iptables stop, Run command/etc/init.d/iptables status

Check the firewall status. Then switch to/usr/hadoop/hadoop-0.20.2/bin, run start-all.sh start service,

This step may not start, this time using the following command to start./start-all.sh, use JPS to view the background Java process,

If you find the following 5 processes, then enter in the browser: http://localhost:50030 and http://localhsot:50070

View the deployed nodes.