The first section of Hadoop Learning: Hadoop configuration Installation

First, configure the virtual machine three

1. Network Configuration

Virtualization software: Vmware workstations 10.7z

Linux system: Centos 6.5-x86_64

Build a small-scale distributed cluster with 4 machines that install Linux environments.

650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M02/7F/54/wKioL1caB6GTPJdWAACjJ08qKpg989.png "title=" 1.png " alt= "Wkiol1cab6gtpjdwaacjj08qkpg989.png"/>

Cluster machine Details

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M00/7F/55/wKiom1caBzqQuGtWAAAg7TrAz7c278.png "title=" 2.png " alt= "Wkiom1cabzqqugtwaaag7traz7c278.png"/>

1. Virtualized network configuration: Bridging network

2.IP address set to fixed IP address

3. Host name configuration, need to modify the place

(1) Configuration Vi/etc/hostname

650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M00/7F/54/wKioL1caCCSgnjDGAAAOVoY1mZo451.png "title=" 3.png " alt= "Wkiol1caccsgnjdgaaaovoy1mzo451.png"/>

(2) Configuration Vi/etc/sysconfig/network

650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M01/7F/55/wKiom1caB3CBVNshAAAYSJP7Mvo857.png "title=" 4.png " alt= "Wkiom1cab3cbvnshaaaysjp7mvo857.png"/>

(3) Configuration vi/etc/hosts

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M01/7F/54/wKioL1caCEvAI1qFAAAdaSBCqzU423.png "title=" 5.png " alt= "Wkiol1cacevai1qfaaadasbcqzu423.png"/>

4. Turn off the firewall

1) effective after reboot

Open: Chkconfig iptables on

OFF: Chkconfig iptables off

2) immediate effect, failure after reboot

Open: Service iptables start

Close: Service iptables stop

After the configuration is complete, the effect:

three nodes ping through each other

host name can be parsed correctly

can ping the outside network normally

650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M00/7F/54/wKioL1caCPaTvMyCAABIeFEhy2w305.png "title=" 6.png " alt= "Wkiol1cacpatvmycaabiefehy2w305.png"/>

2. Java System Environment Configuration

Installation of Java

rpm-qa| grep Java View System-installed Java

Uninstalling the installed Java

Rpm-e–nodeps (package name)

Configuring Environment Variables Vi/etc/profile

#set Java Environment

Export java_home=/usr/java/jdk1.8.0_77

Export JRE_HOME=/USR/JAVA/JDK1.8.0_77/JRE

Export classpath=.: $JAVA _home/lib: $JRE _home/lib: $CLASSPATH

Export path= $JAVA _home/bin: $JRE _home/bin: $PATH

After the configuration is complete, the effect is:

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M02/7F/55/wKiom1caCGHyJd5fAAAf48Z-JKQ416.png "title=" 7.png " alt= "Wkiom1cacghyjd5faaaf48z-jkq416.png"/>

3. No password login between nodes

SSH settings require different operations on the cluster, such as start-up, stop, and distributed daemon shell operations. Authenticating different Hadoop users requires a public/private key pair for Hadoop users to share with different users.

The following command is used to generate a pair using SSH key values. The replication public key forms Id_rsa.pub to the Authorized_keys file and provides read and write permissions to the owner with the Authorized_keys file.

Command: ssh-keygen-t RSA

Command: Cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

After the same command on the other nodes, add the id_rsa.pub content of all the nodes to the Authorized_keys and distribute the Authorized_keys to the/.ssh/directory of all nodes

After the configuration is complete, the effect is:

650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M00/7F/55/wKiom1caCICxosjIAAAnorKqgWc443.png "title=" 8.png " alt= "Wkiom1cacicxosjiaaanorkqgwc443.png"/>

Second, Hadoop configuration installation

The Hadoop version of this document is: hadoop-2.6.4.tar.gz

1. Configure the following on the Namenode node

After you download Hadoop, you can operate a Hadoop cluster in one of the following three support modes:

Local/Standalone mode: Download Hadoop in the system, by default, it will be configured in a standalone mode for running Java programs.

Analog distributed mode: This is a distributed simulation on a single machine. Hadoop daemon Each process, such as HDFs, yarn, MapReduce, etc., will be run as a standalone Java program. This pattern is very useful for development.

Fully distributed mode: This mode is a fully distributed cluster of the smallest two or more computers. We use this model in future chapters.

(1) Change the file Core-site.xml to the following configuration:

<property>

<name>fs.default.name</name>

<value>hdfs://192.168.1.220:9000</value>

</property>

<property>

<name>hadoop.proxyuser.root.hosts</name>

<value>192.168.1.220</value>

</property>

<property>

<name>hadoop.proxyuser.root.groups</name>

<value>*</value>

</property>

(2) Change the file Hdfs-site.xml to the following configuration:

<property>

<name>dfs.name.dir</name>

<value>/usr/hadoop/name</value>

</property>

<property>

<name>dfs.data.dir</name>

<value>/usr/hadoop/data</value>

</property>

<property>

<name>dfs.replication</name>

<value>2</value>

</property>

<property>

<name>dfs.support.append</name>

<value>true</value>

</property>

<property>

<name>dfs.datanode.address</name>

<value>0.0.0.0:50110</value>

</property>

(3) Change the file Mapred-site.xml to the following configuration:

<property>

<name>mapred.job.tracker</name>

<value>http://192.168.1.220:9001</value>

</property>

<property>

<name>mapried.local.dir</name>

<value>/usr/hadoop/var</value>

</property>

(4) Change the file slaves to the following configuration:

Master-hadoop

Slave1-hadoop

Slave2-hadoop

(5) Change the file hadoop-env.sh to the following configuration:

# The Java implementation to use.

Export java_home=/usr/java/jdk1.8.0_77

2. Folders in the creative configuration under/usr/hadooop/

Mkdir tmp

Mkdir logs

Mkdir data

3.hadoop command operation

(1) Use the command HDFs Namenode–format to set the name node as follows

(2) Start command/usr/hadoop/etc/hadoop/start-all.sh

(3) View node status Hadoop dfsadmin-report

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/7F/55/wKiom1caCcbRBdnBAAAfYijr5KI557.png "title=" 11.png "alt=" Wkiom1caccbrbdnbaaafyijr5ki557.png "/>

4. Web node display

http://Master-Hadoop:50070

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M02/7F/54/wKioL1caCg_ySURyAAB9sdFM_ac998.png "title=" 9.png " alt= "Wkiol1cacg_ysuryaab9sdfm_ac998.png"/>

HTTP/master-hadoop:8088

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/7F/54/wKioL1caCiCz7HzsAADNLTGHjUs737.png "title=" 10.png "alt=" Wkiol1cacicz7hzsaadnltghjus737.png "/>

Reference URL:

Http://itindex.net/detail/46949-wordcount

Http://www.cnblogs.com/scotoma/archive/2012/09/18/2689902.html

http://dblab.xmu.edu.cn/blog/install-hadoop-cluster/

Http://192.168.1.200:50070/dfshealth.html#tab-datanode

Http://www.tuicool.com/articles/veim6bU

http://my.oschina.net/u/570654/blog/112780

http://blog.csdn.net/ab198604/article/details/8271860

Http://www.cnblogs.com/shishanyuan/category/709023.html

Http://zhidao.baidu.com/link?url= K6w-swvrs7vtvcg8if1ugq4dbk3e7ychorwa3kjca3d4fie2rydkqnn5h9njcgg7z5hevyu7gyt1jle7-umvmgbroisdtin6mkcvpm2vboa

http://blog.csdn.net/tralonzhang/article/details/7773824

Http://www.cnblogs.com/shishanyuan/p/4177908.html

Http://hadoop.apache.org/docs/r1.0.4/cn/hdfs_user_guide.html

The file list is:

Centos-6.5-x86_64-bin-dvd1.iso

Hadoop-2.6.4.tar.gz

jdk-8u77-linux-x64.rpm

Putty_v0.63.0.0.43510830.exe

This article is from the "If you bloom, the breeze came in" blog, please be sure to keep this source http://iqdutao.blog.51cto.com/2597934/1766864

The first section of Hadoop Learning: Hadoop configuration Installation