hadoop-2.3.0-cdh5.1.0 fully Distributed Build (CentOS-based)

First refer to: "hadoop-2.3.0-cdh5.1.0 pseudo-distributed installation (based on CentOS)"

http://blog.csdn.net/jameshadoop/article/details/39055493

Note: This example uses the root user to build

First, the environment

Operating system: CentOS 6.5 64-bit operating system

Note: Hadoop2.0 above uses the JDK environment is 1.7,linux comes with the JDK to unload, reinstall

Download Address: http://www.oracle.com/technetwork/java/javase/downloads/index.html

Software version: hadoop-2.3.0-cdh5.1.0.tar.gz, zookeeper-3.4.5-cdh5.1.0.tar.gz

Download Address: http://archive.cloudera.com/cdh5/cdh/5/

c1:192.168.58.11

c2:192.168.58.12

c3:192.168.58.13

Second, install the JDK (slightly) see the reference article above III. Configuring Environment variables (Configure environment variables for JDK and Hadoop) iv. System Configuration

1 Shutting down the firewall

Chkconfig iptables off (permanently off)

Configure host name and Hosts file

2, SSH no password authentication configuration

Because Hadoop runs a process that requires remote management of the Hadoop daemon, the Namenode node needs to link each datanode node through SSH (Secure Shell) to stop or start their process, so SSH must be without a password, So we have to make namenode nodes and Datanode nodes into a non-secret communication, the same datanode also need to configure the Namenode node without a password link.

Configure on each machine:

Vi/etc/ssh/sshd_config Open

Rsaauthentication Yes # Enable RSA authentication, pubkeyauthentication Yes # Enable public key private key pairing authentication mode

Master01: Run:ssh-keygen–t rsa–p " do not enter password direct enter

The default is stored in the/root/.ssh directory,

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

[Root@master01. ssh]# ls

Authorized_keys id_rsa id_rsa.pub known_hosts remote copy: SCP authorized_keys c2:~/.ssh/ SCP Authoriz Ed_keys c3:~/.ssh/

v. Configuration of several files (same as each node) 5.1. hadoop/etc/hadoop/hadoop-env.sh Add:

# set to the root ofyour Java installation  
  export Java_home=/usr/java/latest  
   
  # assuming your installation director Y is/usr/local/hadoop  
  Export Hadoop_prefix=/usr/local/hadoop  

5.2. Etc/hadoop/core-site.xml

<configuration>  
    <property>  
         <name>fs.defaultFS</name>  
         <value>hdfs:// c1:9000</value>  
    </property>
     <property>
     <name>hadoop.tmp.dir</name>
     <value>/usr/local/cdh/hadoop/data/tmp</value>
   </property>  
</configuration >

5.3. Etc/hadoop/hdfs-site.xml

<configuration> <property> <!--open Web hdfs--> <name>dfs.webhdfs.enabled</name&   
    Gt   
   <value>true</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.namenode.name.dir</n ame> <value>/usr/local/cdh/hadoop/data/dfs/name</value> <description> namenode Store name T Able (fsimage) local directory (requires modification) </description> </property> <property> <name>dfs.namenode.ed its.dir</name> <value>${dfs.namenode.name.dir}</value> <description>namenode extensive tra Nsactionfile (edits) local directory (requires modification) </description> </property> <property> <name>dfs.da Tanode.data.dir</name> <value>/usr/local/cdh/hadoop/data/dfs/data</value> <descript Ion>datanoDe storage block local directory (requires modification) </description> </property> <property> <name>dfs.permissions</nam e> <value>false</value> </property> <property> <name>dfs.permissions.enable d</name> <value>false</value> </property> </configuration>

5.4 Etc/hadoop/mapred-site.xml

<configuration>  
    <property>  
        <name>mapreduce.framework.name</name>  
        <value >yarn</value>  
    </property>  
</configuration>

5.5 etc/hadoop/yarn-env.sh

# some Java parameters
export java_home=/usr/local/java/jdk1.7.0_67

5.6 Etc/hadoop/yarn-site.xml

<configuration> <property> <name>yarn.resourcemanager.address</name> <value>c1:8032
</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>c1:8030</value> </property> <property> <name>
Yarn.resourcemanager.resource-tracker.address</name> <value>c1:8031</value> </property>
<property> <name>yarn.resourcemanager.admin.address</name> <value>c1:8033</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> &LT;VALUE&GT;C1 :8088</value> </property> <property> <name>yarn.nodemanager.aux-services</name> < value>mapreduce_shuffle</value> </property> <property> <name> Yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value> Org.apache.hadoop.mapred.shufflehandler</value> </property&Gt </configuration>

5.7. Etc/hadoop/slaves

C2
C3

Six: Start and verify the installation is successful

Format: To format HDFs first:

[HTML]View plain copy bin/hdfs Namenode-format boot:

  sbin/start-dfs.sh

  sbin/start-yarn.sh

[Root@c1 hadoop]# JPS
3250 Jps
2491 ResourceManager
2343 Secondarynamenode
2170 NameNode

Datanode node:

[Root@c2 ~]# JPS
4196 Jps
2061 DataNode
2153 NodeManager

[HTML]   View plain copy             1.     Open Browser      NameNode - http://localhost:50070/   2.     Create folders    3.$ bin/hdfs dfs -mkdir /user     $ bin/ Hdfs dfs -mkdir /user/<username>