Because of the need, four virtual machines have been built on their own small Ben, and are planning to build a 1+3 Hadoop distributed system.
Environment: hadoop2.7.0+ubuntu14.04 (64-bit)
First rename each host for the built-in virtual machine
Method: Vi/etc/hostname
Example: Ubuntu0 ubuntu1 ubuntu2 UBUNTU3
I. Configuring the Hosts File
To view the IP of a virtual machine through the ifconfig command, configure the Hosts file
Method: Vi/etc/hosts
192.168.186.XXX ubuntu0
192.168.186.XXX UBUNTU1
192.168.186.XXX UBUNTU2
192.168.186.XXX UBUNTU3
Two. Create a Hadoop run account
Create a Hadoop user group: sudo addgroup Hadoop
Create a Hadoop User: sudo adduser-ingroup hadoop Hadoop
Add Permissions for Hadoop users
Method: sudo vi/etc/sudoers
Add Hadoop all= (all:all) all
Switch Hadoop User: su Hadoop
Three. SSH configuration (master-slave password-free login)
1) Each node generates a public-private key (the generated directory is. SSH)
Ssh-keygen-t Dsa-p '-F ~/.ssh/ID_DSACD sshcat id_dsa.pub >> authorized_keys
Standalone test password-free login: SSH localhost (or host name)
Exit Command: Exit
2) Let the master node log in to the child node via SSH
SCP [email protected]:~/.ssh/id_dsa.pub./master_dsa.pubcat master_dsa.pub >> authorized_keys
Repeat the appeal action on the child node.
Four. Download and unzip the Hadoop installation package
1 Installing the Java Environment
sudo apt-get install OPENJDK-7-JDK
View installation results: Java-version
2 Downloads hadoop2.7.0
Unzip: sudo tar xzf hadoop-2.7.0.tar.gz
Unzip Hadoop under/usr/local/hadoop:
sudo mv Hadoop-2.7.0/usr/local/hadoop
Modify permissions: sudo chmod 777/usr/local/hadoop
3 Configuring ~/.BASHRC
To view the installation path for Java: Update-alternatives--config java
Configure the. bashrc file: VI ~/.BASHRC Add at the end:
#HADOOP VARIABLES startexport java_home=/usr/lib/jvm/java-7-openjdk-i386export hadoop_install=/usr/local/ Hadoopexport path= $PATH: $HADOOP _install/binexport path= $PATH: $HADOOP _install/sbinexporthadoop_mapred_ Home=$HADOOP _installexport hadoop_common_home=$HADOOP _installexport hadoop_hdfs_home=$HADOOP _ Installexport yarn_home=$HADOOP _installexport hadoop_common_lib_native_dir= $HADOOP _install/lib/ Nativeexport hadoop_opts= "-djava.library.path= $HADOOP _install/lib"#HADOOP VARIABLES END
Executes the source ~/.BASHRC, making the added environment variable effective.
4. Edit/usr/local/hadoop/etc/hadoop/hadoop-env.sh
Add to
Export java_home=/usr/lib/jvm/java-7-openjdk-amd64export hadoop_home=/usr/local/hadoopexport YARN_HOME=/ usr/local/hadoopexport hadoop_conf_dir= $HADOOP _home/etc/hadoopexport yarn_conf_dir= $HADOOP _home/etc/ Hadoop
5. Edit/usr/local/hadoop/etc/hadoop/yarn-env.sh
Export yarn_conf_dir= "${yarn_conf_dir:-$HADOOP _yarn_home/etc/hadoop}"export java_home=/usr/lib/jvm/ Java-7-openjdk-amd64
Five. Configure Namenode, modify the site file (/usr/local/hadoop/etc/hadoop/)
1, Configuration Core-site.xml
<property><name>fs.default.name</name><value>hdfs://localhost:9000</value></ Property>
2, Yarn-site.xml
<property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value ></property><property><name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</ Name><value>org.apache.hadoop.mapred.shufflehandler</value></property>
3, create Mapred-site.xml,
CP Mapred-site.xml.template Mapred-site.xml
and add
<property><name>mapreduce.framework.name</name><value>yarn</value></property >
4, Configuration Hdfs-site.xml
cd/usr/local/hadoop/mkdir hdfsmkdir hdfs/datamkdir hdfs/name
Edit Open Hdfs-site.xml
<property><name>dfs.replication</name><value>1</value></property>< property><name>dfs.namenode.name.dir</name><value>file:/usr/local/hadoop/hdfs/name</ Value></property><property><name>dfs.datanode.data.dir</name><value>file:/usr /local/hadoop/hdfs/data</value></property>
Six. Configuring Masters and Slaves Files
Fill in the appropriate host name in two files
Seven. Replicating Hadoop to a node
Scp-r./hadoop ubuntu1:~scp-r./hadoop ubuntu2:~ scp-r. /hadoop ubuntu3:~
Viii. Formatting Namenode
Note: "Successfully formatted" appears above as a success.
Nine. Start Hadoop
Browser opens http://localhost:50070/, you will see the HDFs administration page
Browser opens http://localhost:8088, you will see the Hadoop Process Management page
hadoop2.7.0 Distributed System Setup (ubuntu14.04)