Hadoop-2.X installation and configuration

Hadoop-2.X installation and configuration

We use a single-node cluster as an example to demonstrate how to install Hadoop2.6.0. The installation of ssh and jdk is described in the previous article.

Installation steps:

(1) Place the downloaded Hadoop installation package to the specified directory, for example, to the home Directory of your current user. Run the following command to decompress the installation package:

Tar xzf hadoop-2.6.0.tar.gz

(2) EDIT. bashrc in the home directory and add the following items:

 

# set Hadoop Enivorment for version 2.6.0#export HADOOP=/Users/bowen/Software/hadoop-1.2.1export HADOOP_HOME=/Users/bowen/Software/hadoop-2.6.0export PATH=$PATH:$HADOOP_HOME/binexport HADOOP_MAPRED_HOME=$HADOOP_HOMEexport HADOOP_COMMON_HOME=$HADOOP_HOMEexport HADOOP_HDFS_HOME=$HADOOP_HOMEexport HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexport YARN_HOME=$HADOOP_HOME

This configuration also comes from the Internet. Hadoop2.x is much different from Hadoop1.x.

 

(3) add namenode and datanode Directories

 

mkdir -p $HADOOP_HOME/yarn/yarn_data/hdfs/namenodemkdir -p $HADOOP_HOME/yarn/yarn_data/hdfs/datanode

Note: Replace $ HADOOP_HOME with the hadoop installation directory.

 

(4) edit the yarn-site.xml and add the following Configuration:

 

<property>  <name>yarn.nodemanager.aux-services</name>  <value>mapreduce_shuffle</value></property><property>  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>  <value>org.apache.hadoop.mapred.ShuffleHandler</value></property>

Note: You can change the port number and local file directory as needed. If you use a local file directory, you are advised to create these directories in advance.

 

For example, in the configuration file above, we use the/home/user/hadoop-2.6.0/nodemanager-local-dirs directory.

 

(5) edit the core-site.xml and add the following Configuration:

 

<property>    <name>fs.defaultFS</name>    <value>hdfs://localhost:9000</value> </property>

(6) edit the hdfs-site.xml and add the following Configuration:

 

 

 <property>    <name>dfs.replication</name>    <value>1</value>  </property>  <property>    <name>dfs.namenode.name.dir</name>    <value>/home/user/hadoop-2.2.0/yarn/yarn_data/hdfs/namenode</value>  </property>  <property>    <name>dfs.datanode.data.dir</name>    <value>/home/user/hadoop-2.2.0/yarn/yarn_data/hdfs/datanode</value>  </property>

 

(7) edit the mapred-site.xml and add the following Configuration:

 

  <property>  <name>mapreduce.cluster.temp.dir</name>    <value></value>    <description>No description</description>    <final>true</final>  </property>  <property><name>mapreduce.cluster.local.dir</name><value></value>    <description>No description</description>    <final>true</final>  </property>

Note: If this file does not exist, you can get it by copying the mapred-site.xml.template file.

 

(8) edit the hodoop-env.sh and modify the following behavior:

 

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.6.0.jdk/Contents/Home

Note: Make sure that JAVA is correctly installed. If not, you can refer to the relevant documentation for JAVA installation.

 

(9) initialize noamenode

Bin/hadoop namenode-format

(10) Start the hadoop daemon. Run the following commands in sequence:

 

sbin/hadoop-daemon.sh start namenodesbin/hadoop-daemon.sh start datanodesbin/hadoop-daemon.sh start secondarynamenodesbin/yarn-daemon.sh start resourcemanagersbin/yarn-daemon.sh start nodemanagersbin/mr-jobhistory-daemon.sh start historyserver

(11) enter the following URL in the browser to view the status of each service:

 

 

Http: // localhost: 50070

Http: // localhost: 8088

Http: // localhost: 19888

12) to stop the above services, run the following commands in sequence:

 

sbin/hadoop-daemon.sh stop namenodesbin/hadoop-daemon.sh stop datanodesbin/hadoop-daemon.sh stop secondarynamenodesbin/yarn-daemon.sh stop resourcemanagersbin/yarn-daemon.sh stop nodemanagersbin/mr-jobhistory-daemon.sh stop historyserver

Note: You can place the preceding commands in a script file to facilitate execution.

 

13) test

Run the following command to create a data file and copy it to dfs:

Touch in

Nano in

Hadoop is fast

Hadoop is cool

Bin/hadoop fs-copyFromLocal in // in

Use the wordcount provided by Hadoop for testing:

Bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount/in/out

If the installation is correct, you can view the processing information after the task is called on the screen.

If you call this task again, an error message may appear in the/out directory. You can call the following command to clear the corresponding directory, for example:

Bin/hadoop fs-rm-r/out