hadoop2.7.0 Distributed System Setup (ubuntu14.04)

Because of the need, four virtual machines have been built on their own small Ben, and are planning to build a 1+3 Hadoop distributed system.

Environment: hadoop2.7.0+ubuntu14.04 (64-bit)

First rename each host for the built-in virtual machine

Method: Vi/etc/hostname

Example: Ubuntu0 ubuntu1 ubuntu2 UBUNTU3

I. Configuring the Hosts File

To view the IP of a virtual machine through the ifconfig command, configure the Hosts file

Method: Vi/etc/hosts

192.168.186.XXX ubuntu0

192.168.186.XXX UBUNTU1

192.168.186.XXX UBUNTU2

192.168.186.XXX UBUNTU3

Two. Create a Hadoop run account

Create a Hadoop user group: sudo addgroup Hadoop

Create a Hadoop User: sudo adduser-ingroup hadoop Hadoop

Add Permissions for Hadoop users

Method: sudo vi/etc/sudoers

Add Hadoop all= (all:all) all

Switch Hadoop User: su Hadoop

Three. SSH configuration (master-slave password-free login)

1) Each node generates a public-private key (the generated directory is. SSH)

Ssh-keygen-t Dsa-p '-F ~/.ssh/ID_DSACD sshcat id_dsa.pub >> authorized_keys

Standalone test password-free login: SSH localhost (or host name)

Exit Command: Exit

2) Let the master node log in to the child node via SSH

SCP [email protected]:~/.ssh/id_dsa.pub./master_dsa.pubcat master_dsa.pub >> authorized_keys

Repeat the appeal action on the child node.

Four. Download and unzip the Hadoop installation package

1 Installing the Java Environment

sudo apt-get install OPENJDK-7-JDK

View installation results: Java-version

2 Downloads hadoop2.7.0

Unzip: sudo tar xzf hadoop-2.7.0.tar.gz

Unzip Hadoop under/usr/local/hadoop:

sudo mv Hadoop-2.7.0/usr/local/hadoop

Modify permissions: sudo chmod 777/usr/local/hadoop

3 Configuring ~/.BASHRC

To view the installation path for Java: Update-alternatives--config java

Configure the. bashrc file: VI ~/.BASHRC Add at the end:

#HADOOP VARIABLES startexport java_home=/usr/lib/jvm/java-7-openjdk-i386export hadoop_install=/usr/local/  Hadoopexport path= $PATH: $HADOOP _install/binexport path= $PATH: $HADOOP _install/sbinexporthadoop_mapred_ Home=$HADOOP _installexport hadoop_common_home=$HADOOP _installexport hadoop_hdfs_home=$HADOOP _ Installexport yarn_home=$HADOOP _installexport hadoop_common_lib_native_dir= $HADOOP _install/lib/  Nativeexport hadoop_opts= "-djava.library.path= $HADOOP _install/lib"#HADOOP VARIABLES END    

Executes the source ~/.BASHRC, making the added environment variable effective.

4. Edit/usr/local/hadoop/etc/hadoop/hadoop-env.sh

Add to

Export java_home=/usr/lib/jvm/java-7-openjdk-amd64export hadoop_home=/usr/local/hadoopexport YARN_HOME=/ usr/local/hadoopexport hadoop_conf_dir= $HADOOP _home/etc/hadoopexport yarn_conf_dir= $HADOOP _home/etc/ Hadoop   

5. Edit/usr/local/hadoop/etc/hadoop/yarn-env.sh

Export yarn_conf_dir= "${yarn_conf_dir:-$HADOOP _yarn_home/etc/hadoop}"export java_home=/usr/lib/jvm/ Java-7-openjdk-amd64

Five. Configure Namenode, modify the site file (/usr/local/hadoop/etc/hadoop/)

1, Configuration Core-site.xml

<property><name>fs.default.name</name><value>hdfs://localhost:9000</value></ Property>

2, Yarn-site.xml

<property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value ></property><property><name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</ Name><value>org.apache.hadoop.mapred.shufflehandler</value></property>

3, create Mapred-site.xml,

CP Mapred-site.xml.template Mapred-site.xml

and add

<property><name>mapreduce.framework.name</name><value>yarn</value></property >

4, Configuration Hdfs-site.xml

cd/usr/local/hadoop/mkdir hdfsmkdir hdfs/datamkdir hdfs/name 

Edit Open Hdfs-site.xml

<property><name>dfs.replication</name><value>1</value></property>< property><name>dfs.namenode.name.dir</name><value>file:/usr/local/hadoop/hdfs/name</ Value></property><property><name>dfs.datanode.data.dir</name><value>file:/usr /local/hadoop/hdfs/data</value></property>

Six. Configuring Masters and Slaves Files

Fill in the appropriate host name in two files

Seven. Replicating Hadoop to a node

Scp-r./hadoop ubuntu1:~scp-r./hadoop ubuntu2:~ scp-r.  /hadoop ubuntu3:~

Viii. Formatting Namenode

Note: "Successfully formatted" appears above as a success.

Nine. Start Hadoop

Browser opens http://localhost:50070/, you will see the HDFs administration page

Browser opens http://localhost:8088, you will see the Hadoop Process Management page

hadoop2.7.0 Distributed System Setup (ubuntu14.04)