Build and summarize the Hadoop2.2.0/2.3.0 environment on 64-bit Ubuntu 12.04 Server

Part 1 preparations for Hadoop 2.2.0 Construction)

Hadoop: download the latest version Hadoop2.2.0 from the Apache official website. Currently, the linux32-bit system executable file is provided officially. to deploy the file on a 64-bit system, you need to download the src Source Code and compile it by yourself.

: Http://apache.claz.org/hadoop/common/hadoop-2.2.0/

OS of three hosts: Ubuntu 12.04-64 server

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition

Build a Hadoop environment (using virtual machines to build two Ubuntu systems in a Winodws environment)

L

Select the location where you want to install java, such as in the/usr/directory, and create a folder named java (mkdirjava)

Move the jdk-8-linux-x64.tar.gz file to/usr/java

Decompress: tar -zxvfjdk-8-linux-x64.tar.gz

Now, the jkd installation and preparation are complete, and the environment variables are configured below

4.3 open/etc/profile (vim/etc/profile)

Add the following content at the end:

ExportJAVA_HOME =/usr/java/jdk1.8.0/

ExportJRE_HOME =/usr/java/jdk1.8.0/

Export PATH = $ JAVA_HOME/bin: $ JAVA_HOME/jre/bin: $ PATH

ExportCLASSPATH = $ CLASSPATH:.: $ JAVA_HOME/lib: $ JAVA_HOME/jre/lib

ExportJAVA_BIN =/usr/java/jdk1.8.0/bin

Run source/etc/profile

Verify whether the installation is successful: java-version

Java version "1.8.0"

Java (TM) SERuntime Environment (build 1.8.0-b132)

JavaHotSpot (TM) 64-Bit Server VM (build 25.0-b70, mixed mode)

NOTE: If each machine performs the same operation, it is best to install java in the same path (not required, but this will make subsequent configuration much more convenient)

Ufw disable (restart takes effect)

Now, the preparation is complete!

Part 2 Cluster Environment Construction

1. Here we build a cluster composed of three machines:

172.109120.200 haduser/passwd master namenode/secondarynamenode/resourcemanger

172.120.120.201 haduser/passwd slave1 datanode/nodemanage

172.128.120.202 haduser/passwd slave2 datanode/nodemanage

The above columns are IP, user/passwd, hostname, and roles in the cluster.

Enter the hadoop-2.2.0 Installation:

Follow these steps to log on with haduser.

Since the configuration on each machine in the hadoop cluster is basically the same, we first configure and deploy the configuration on namenode and then copy it to other nodes. Therefore, the installation process is equivalent to running on each machine. However, pay attention to the 64-bit and 32-bit systems in the cluster.

1. decompress the file

Decompress hadoop-2.2.0_64.tar.gz downloaded in step 1 to/home/haduser/Hadoop. Store it elsewhere for backup. Note: The installation path for each machine must be the same !!

2. hadoop configuration process

Seven configuration files are involved:

~ Hadoop-2.2.0/etc/hadoop/hadoop-env.sh

~ Hadoop-2.2.0/etc/hadoop/yarn-env.sh

~ /Hadoop-2.2.0/etc/hadoop/slaves

~ Hadoop-2.2.0/etc/hadoop/core-site.xml

~ Hadoop-2.2.0/etc/hadoop/hdfs-site.xml

~ Hadoop-2.2.0/etc/hadoop/mapred-site.xml

~ Hadoop-2.2.0/etc/hadoop/yarn-site.xml

If the preceding files do not exist by default, you can copy the corresponding template file to obtain them.

Profile 1: hadoop-env.sh

Modify JAVA_HOME value export JAVA_HOME =/usr/java/jdk1.8.0

Profile 2: yarn-env.sh

Modify JAVA_HOME value exportJAVA_HOME =/usr/java/jdk1.8.0

Configuration File 3: slaves (this file stores all slave nodes)

Write the following content:

Slave1

Slave2

1 2 3 Next Page