hadoop-2.x Installation and Configuration

For example, we demonstrate how to install Hadoop2.6.0 in a single node cluster. The installation of SSH and JDK is described in the previous article and is not covered here.

Installation steps:

(1) Place the downloaded Hadoop installation package in the specified directory, such as the home directory of your current user. Execute the following command to unpack the installation package:

Tar xzf hadoop-2.6.0.tar.gz

(2) Edit the. BASHRC in the home directory and add the following:

# set Hadoop enivorment for version 2.6.0#export Hadoop=/users/bowen/software/hadoop-1.2.1export hadoop_home=/users/ Bowen/software/hadoop-2.6.0export path= $PATH: $HADOOP _home/binexport hadoop_mapred_home= $HADOOP _homeexport HADOOP_ Common_home= $HADOOP _homeexport hadoop_hdfs_home= $HADOOP _homeexport hadoop_conf_dir= $HADOOP _home/etc/hadoopexport Yarn_home= $HADOOP _home

This configuration is also referenced on the internet, hadoop2.x compared to hadoop1.x difference is still relatively large.

(3) Add Namenode and Datanode directories

Mkdir-p $HADOOP _home/yarn/yarn_data/hdfs/namenodemkdir-p $HADOOP _home/yarn/yarn_data/hdfs/datanode

Note: Replace the $hadoop_home with the installation directory for HADOOP

(4) Edit Yarn-site.xml, add the following configuration:

<property>  <name>yarn.nodemanager.aux-services</name>  <value>mapreduce_shuffle </value></property><property>  <name> Yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>  <value> Org.apache.hadoop.mapred.shufflehandler</value></property>

Note: Depending on your needs, you can change the port number and the local file directory. If you are using a local file directory, it is best to create these corresponding directories beforehand.

For example, in the above configuration file, we used the/home/user/hadoop-2.6.0/nodemanager-local-dirs directory.

(5) Edit Core-site.xml, add the following configuration:

<property>    <name>fs.defaultFS</name>    <value>hdfs://localhost:9000</value > </property>

(6) Edit Hdfs-site.xml, add the following configuration:

<property>    <name>dfs.replication</name>    <value>1</value>  </ property>  <property>    <name>dfs.namenode.name.dir</name>    <value>/home/ user/hadoop-2.2.0/yarn/yarn_data/hdfs/namenode</value>  </property>  <property>    <name>dfs.datanode.data.dir</name>    <value>/home/user/hadoop-2.2.0/yarn/yarn_data/hdfs/ Datanode</value>  </property>

(7) Edit Mapred-site.xml, add the following configuration:

  <property>  <name>mapreduce.cluster.temp.dir</name>    <value></value>    <description>no description</description>    <final>true</final>  </property>  <property><name>mapreduce.cluster.local.dir</name><value></value>    < Description>no description</description>    <final>true</final>  </property>

Note: If the file does not exist, it can be obtained by copying the mapred-site.xml.template file.

(8) Edit hodoop-env.sh, modify the following corresponding behavior:

Export Java_home=/library/java/javavirtualmachines/jdk1.6.0.jdk/contents/home

Note: Make sure that Java is installed correctly. If not, you can install Java by referencing the relevant documentation.

(9) Initialize Noamenode

Bin/hadoop Namenode-format

(10) Start the Hadoop daemon. Execute the following command in turn:

sbin/hadoop-daemon.sh start namenodesbin/hadoop-daemon.sh start datanodesbin/hadoop-daemon.sh start secondarynamenodesbin/yarn-daemon.sh start resourcemanagersbin/yarn-daemon.sh start nodemanagersbin/ mr-jobhistory-daemon.sh Start Historyserver

(11) Enter the following URL in the browser to view the status of each service:

http://localhost:50070

http://localhost:8088

http://localhost:19888

12) If you need to stop the above services, you can run the following command in turn:

sbin/hadoop-daemon.sh Stop namenodesbin/hadoop-daemon.sh Stop datanodesbin/hadoop-daemon.sh stop secondarynamenodesbin/yarn-daemon.sh Stop resourcemanagersbin/yarn-daemon.sh Stop nodemanagersbin/ mr-jobhistory-daemon.sh Stop Historyserver

Note: You can put more than one of these commands into a script file for easy execution.

13) Testing

Run the following command to create the data file and copy it to DFS:

Touch in

Nano in

Hadoop is fast

Hadoop is cool

Bin/hadoop fs-copyfromlocal in//in

Test with the wordcount that comes with Hadoop:

Bin/hadoop jar Share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar Wordcount/in/out

If the installation is correct, you will be able to see the processing information on the screen after the task is called.

If you call the task again, you may receive an error message that already exists in the/out directory, at which point you can call the following command to clear the corresponding directory, such as:

Bin/hadoop Fs-rm-r/Out

hadoop-2.x Installation and Configuration