First, configure the virtual machine three
1. Network Configuration
Virtualization software: Vmware workstations 10.7z
Linux system: Centos 6.5-x86_64
Build a small-scale distributed cluster with 4 machines that install Linux environments.
650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M02/7F/54/wKioL1caB6GTPJdWAACjJ08qKpg989.png "title=" 1.png " alt= "Wkiol1cab6gtpjdwaacjj08qkpg989.png"/>
Cluster machine Details
650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M00/7F/55/wKiom1caBzqQuGtWAAAg7TrAz7c278.png "title=" 2.png " alt= "Wkiom1cabzqqugtwaaag7traz7c278.png"/>
1. Virtualized network configuration: Bridging network
2.IP address set to fixed IP address
3. Host name configuration, need to modify the place
(1) Configuration Vi/etc/hostname
650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M00/7F/54/wKioL1caCCSgnjDGAAAOVoY1mZo451.png "title=" 3.png " alt= "Wkiol1caccsgnjdgaaaovoy1mzo451.png"/>
(2) Configuration Vi/etc/sysconfig/network
650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M01/7F/55/wKiom1caB3CBVNshAAAYSJP7Mvo857.png "title=" 4.png " alt= "Wkiom1cab3cbvnshaaaysjp7mvo857.png"/>
(3) Configuration vi/etc/hosts
650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M01/7F/54/wKioL1caCEvAI1qFAAAdaSBCqzU423.png "title=" 5.png " alt= "Wkiol1cacevai1qfaaadasbcqzu423.png"/>
4. Turn off the firewall
1) effective after reboot
Open: Chkconfig iptables on
OFF: Chkconfig iptables off
2) immediate effect, failure after reboot
Open: Service iptables start
Close: Service iptables stop
After the configuration is complete, the effect:
three nodes ping through each other
host name can be parsed correctly
can ping the outside network normally
650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M00/7F/54/wKioL1caCPaTvMyCAABIeFEhy2w305.png "title=" 6.png " alt= "Wkiol1cacpatvmycaabiefehy2w305.png"/>
2. Java System Environment Configuration
Installation of Java
rpm-qa| grep Java View System-installed Java
Uninstalling the installed Java
Rpm-e–nodeps (package name)
Configuring Environment Variables Vi/etc/profile
#set Java Environment
Export java_home=/usr/java/jdk1.8.0_77
Export JRE_HOME=/USR/JAVA/JDK1.8.0_77/JRE
Export classpath=.: $JAVA _home/lib: $JRE _home/lib: $CLASSPATH
Export path= $JAVA _home/bin: $JRE _home/bin: $PATH
After the configuration is complete, the effect is:
650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M02/7F/55/wKiom1caCGHyJd5fAAAf48Z-JKQ416.png "title=" 7.png " alt= "Wkiom1cacghyjd5faaaf48z-jkq416.png"/>
3. No password login between nodes
SSH settings require different operations on the cluster, such as start-up, stop, and distributed daemon shell operations. Authenticating different Hadoop users requires a public/private key pair for Hadoop users to share with different users.
The following command is used to generate a pair using SSH key values. The replication public key forms Id_rsa.pub to the Authorized_keys file and provides read and write permissions to the owner with the Authorized_keys file.
Command: ssh-keygen-t RSA
Command: Cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
After the same command on the other nodes, add the id_rsa.pub content of all the nodes to the Authorized_keys and distribute the Authorized_keys to the/.ssh/directory of all nodes
After the configuration is complete, the effect is:
650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M00/7F/55/wKiom1caCICxosjIAAAnorKqgWc443.png "title=" 8.png " alt= "Wkiom1cacicxosjiaaanorkqgwc443.png"/>
Second, Hadoop configuration installation
The Hadoop version of this document is: hadoop-2.6.4.tar.gz
1. Configure the following on the Namenode node
After you download Hadoop, you can operate a Hadoop cluster in one of the following three support modes:
Local/Standalone mode: Download Hadoop in the system, by default, it will be configured in a standalone mode for running Java programs.
Analog distributed mode: This is a distributed simulation on a single machine. Hadoop daemon Each process, such as HDFs, yarn, MapReduce, etc., will be run as a standalone Java program. This pattern is very useful for development.
Fully distributed mode: This mode is a fully distributed cluster of the smallest two or more computers. We use this model in future chapters.
(1) Change the file Core-site.xml to the following configuration:
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.1.220:9000</value>
</property>
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>192.168.1.220</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
(2) Change the file Hdfs-site.xml to the following configuration:
<property>
<name>dfs.name.dir</name>
<value>/usr/hadoop/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/hadoop/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.support.append</name>
<value>true</value>
</property>
<property>
<name>dfs.datanode.address</name>
<value>0.0.0.0:50110</value>
</property>
(3) Change the file Mapred-site.xml to the following configuration:
<property>
<name>mapred.job.tracker</name>
<value>http://192.168.1.220:9001</value>
</property>
<property>
<name>mapried.local.dir</name>
<value>/usr/hadoop/var</value>
</property>
(4) Change the file slaves to the following configuration:
Master-hadoop
Slave1-hadoop
Slave2-hadoop
(5) Change the file hadoop-env.sh to the following configuration:
# The Java implementation to use.
Export java_home=/usr/java/jdk1.8.0_77
2. Folders in the creative configuration under/usr/hadooop/
Mkdir tmp
Mkdir logs
Mkdir data
3.hadoop command operation
(1) Use the command HDFs Namenode–format to set the name node as follows
(2) Start command/usr/hadoop/etc/hadoop/start-all.sh
(3) View node status Hadoop dfsadmin-report
650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/7F/55/wKiom1caCcbRBdnBAAAfYijr5KI557.png "title=" 11.png "alt=" Wkiom1caccbrbdnbaaafyijr5ki557.png "/>
4. Web node display
http://Master-Hadoop:50070
650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M02/7F/54/wKioL1caCg_ySURyAAB9sdFM_ac998.png "title=" 9.png " alt= "Wkiol1cacg_ysuryaab9sdfm_ac998.png"/>
HTTP/master-hadoop:8088
650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/7F/54/wKioL1caCiCz7HzsAADNLTGHjUs737.png "title=" 10.png "alt=" Wkiol1cacicz7hzsaadnltghjus737.png "/>
Reference URL:
Http://itindex.net/detail/46949-wordcount
Http://www.cnblogs.com/scotoma/archive/2012/09/18/2689902.html
http://dblab.xmu.edu.cn/blog/install-hadoop-cluster/
Http://192.168.1.200:50070/dfshealth.html#tab-datanode
Http://www.tuicool.com/articles/veim6bU
http://my.oschina.net/u/570654/blog/112780
http://blog.csdn.net/ab198604/article/details/8271860
Http://www.cnblogs.com/shishanyuan/category/709023.html
Http://zhidao.baidu.com/link?url= K6w-swvrs7vtvcg8if1ugq4dbk3e7ychorwa3kjca3d4fie2rydkqnn5h9njcgg7z5hevyu7gyt1jle7-umvmgbroisdtin6mkcvpm2vboa
http://blog.csdn.net/tralonzhang/article/details/7773824
Http://www.cnblogs.com/shishanyuan/p/4177908.html
Http://hadoop.apache.org/docs/r1.0.4/cn/hdfs_user_guide.html
The file list is:
Centos-6.5-x86_64-bin-dvd1.iso
Hadoop-2.6.4.tar.gz
jdk-8u77-linux-x64.rpm
Putty_v0.63.0.0.43510830.exe
This article is from the "If you bloom, the breeze came in" blog, please be sure to keep this source http://iqdutao.blog.51cto.com/2597934/1766864
The first section of Hadoop Learning: Hadoop configuration Installation