The first section of Hadoop Learning: Hadoop configuration Installation

Source: Internet
Author: User




First, configure the virtual machine three


1. Network Configuration


Virtualization software: Vmware workstations 10.7z

Linux system: Centos 6.5-x86_64



Build a small-scale distributed cluster with 4 machines that install Linux environments.


650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M02/7F/54/wKioL1caB6GTPJdWAACjJ08qKpg989.png "title=" 1.png " alt= "Wkiol1cab6gtpjdwaacjj08qkpg989.png"/>




Cluster machine Details


650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M00/7F/55/wKiom1caBzqQuGtWAAAg7TrAz7c278.png "title=" 2.png " alt= "Wkiom1cabzqqugtwaaag7traz7c278.png"/>




1. Virtualized network configuration: Bridging network

2.IP address set to fixed IP address

3. Host name configuration, need to modify the place


(1) Configuration Vi/etc/hostname


650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M00/7F/54/wKioL1caCCSgnjDGAAAOVoY1mZo451.png "title=" 3.png " alt= "Wkiol1caccsgnjdgaaaovoy1mzo451.png"/>

(2) Configuration Vi/etc/sysconfig/network


650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M01/7F/55/wKiom1caB3CBVNshAAAYSJP7Mvo857.png "title=" 4.png " alt= "Wkiom1cab3cbvnshaaaysjp7mvo857.png"/>

(3) Configuration vi/etc/hosts


650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M01/7F/54/wKioL1caCEvAI1qFAAAdaSBCqzU423.png "title=" 5.png " alt= "Wkiol1cacevai1qfaaadasbcqzu423.png"/>



4. Turn off the firewall


1) effective after reboot


Open: Chkconfig iptables on

OFF: Chkconfig iptables off



2) immediate effect, failure after reboot


Open: Service iptables start

Close: Service iptables stop


After the configuration is complete, the effect:


three nodes ping through each other

host name can be parsed correctly

can ping the outside network normally

650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M00/7F/54/wKioL1caCPaTvMyCAABIeFEhy2w305.png "title=" 6.png " alt= "Wkiol1cacpatvmycaabiefehy2w305.png"/>


2. Java System Environment Configuration


Installation of Java


rpm-qa| grep Java View System-installed Java


Uninstalling the installed Java


Rpm-e–nodeps (package name)



Configuring Environment Variables Vi/etc/profile


#set Java Environment

Export java_home=/usr/java/jdk1.8.0_77

Export JRE_HOME=/USR/JAVA/JDK1.8.0_77/JRE

Export classpath=.: $JAVA _home/lib: $JRE _home/lib: $CLASSPATH

Export path= $JAVA _home/bin: $JRE _home/bin: $PATH


After the configuration is complete, the effect is:


650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M02/7F/55/wKiom1caCGHyJd5fAAAf48Z-JKQ416.png "title=" 7.png " alt= "Wkiom1cacghyjd5faaaf48z-jkq416.png"/>





3. No password login between nodes


SSH settings require different operations on the cluster, such as start-up, stop, and distributed daemon shell operations. Authenticating different Hadoop users requires a public/private key pair for Hadoop users to share with different users.


The following command is used to generate a pair using SSH key values. The replication public key forms Id_rsa.pub to the Authorized_keys file and provides read and write permissions to the owner with the Authorized_keys file.



Command: ssh-keygen-t RSA

Command: Cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys


After the same command on the other nodes, add the id_rsa.pub content of all the nodes to the Authorized_keys and distribute the Authorized_keys to the/.ssh/directory of all nodes


After the configuration is complete, the effect is:

650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M00/7F/55/wKiom1caCICxosjIAAAnorKqgWc443.png "title=" 8.png " alt= "Wkiom1cacicxosjiaaanorkqgwc443.png"/>




Second, Hadoop configuration installation



The Hadoop version of this document is: hadoop-2.6.4.tar.gz


1. Configure the following on the Namenode node


After you download Hadoop, you can operate a Hadoop cluster in one of the following three support modes:


Local/Standalone mode: Download Hadoop in the system, by default, it will be configured in a standalone mode for running Java programs.


Analog distributed mode: This is a distributed simulation on a single machine. Hadoop daemon Each process, such as HDFs, yarn, MapReduce, etc., will be run as a standalone Java program. This pattern is very useful for development.


Fully distributed mode: This mode is a fully distributed cluster of the smallest two or more computers. We use this model in future chapters.




(1) Change the file Core-site.xml to the following configuration:


<property>

<name>fs.default.name</name>

<value>hdfs://192.168.1.220:9000</value>

</property>

<property>

<name>hadoop.proxyuser.root.hosts</name>

<value>192.168.1.220</value>

</property>

<property>

<name>hadoop.proxyuser.root.groups</name>

<value>*</value>

</property>


(2) Change the file Hdfs-site.xml to the following configuration:


<property>

<name>dfs.name.dir</name>

<value>/usr/hadoop/name</value>

</property>


<property>

<name>dfs.data.dir</name>

<value>/usr/hadoop/data</value>

</property>


<property>

<name>dfs.replication</name>

<value>2</value>

</property>


<property>

<name>dfs.support.append</name>

<value>true</value>

</property>


<property>

<name>dfs.datanode.address</name>

<value>0.0.0.0:50110</value>

</property>



(3) Change the file Mapred-site.xml to the following configuration:


<property>

<name>mapred.job.tracker</name>

<value>http://192.168.1.220:9001</value>

</property>


<property>

<name>mapried.local.dir</name>

<value>/usr/hadoop/var</value>

</property>



(4) Change the file slaves to the following configuration:


Master-hadoop


Slave1-hadoop


Slave2-hadoop



(5) Change the file hadoop-env.sh to the following configuration:


# The Java implementation to use.

Export java_home=/usr/java/jdk1.8.0_77






2. Folders in the creative configuration under/usr/hadooop/


Mkdir tmp

Mkdir logs

Mkdir data





3.hadoop command operation

(1) Use the command HDFs Namenode–format to set the name node as follows


(2) Start command/usr/hadoop/etc/hadoop/start-all.sh


(3) View node status Hadoop dfsadmin-report


650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/7F/55/wKiom1caCcbRBdnBAAAfYijr5KI557.png "title=" 11.png "alt=" Wkiom1caccbrbdnbaaafyijr5ki557.png "/>




4. Web node display


http://Master-Hadoop:50070

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M02/7F/54/wKioL1caCg_ySURyAAB9sdFM_ac998.png "title=" 9.png " alt= "Wkiol1cacg_ysuryaab9sdfm_ac998.png"/>




HTTP/master-hadoop:8088



650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/7F/54/wKioL1caCiCz7HzsAADNLTGHjUs737.png "title=" 10.png "alt=" Wkiol1cacicz7hzsaadnltghjus737.png "/>










Reference URL:


Http://itindex.net/detail/46949-wordcount

Http://www.cnblogs.com/scotoma/archive/2012/09/18/2689902.html

http://dblab.xmu.edu.cn/blog/install-hadoop-cluster/

Http://192.168.1.200:50070/dfshealth.html#tab-datanode

Http://www.tuicool.com/articles/veim6bU

http://my.oschina.net/u/570654/blog/112780

http://blog.csdn.net/ab198604/article/details/8271860

Http://www.cnblogs.com/shishanyuan/category/709023.html

Http://zhidao.baidu.com/link?url= K6w-swvrs7vtvcg8if1ugq4dbk3e7ychorwa3kjca3d4fie2rydkqnn5h9njcgg7z5hevyu7gyt1jle7-umvmgbroisdtin6mkcvpm2vboa

http://blog.csdn.net/tralonzhang/article/details/7773824

Http://www.cnblogs.com/shishanyuan/p/4177908.html

Http://hadoop.apache.org/docs/r1.0.4/cn/hdfs_user_guide.html



The file list is:


Centos-6.5-x86_64-bin-dvd1.iso

Hadoop-2.6.4.tar.gz

jdk-8u77-linux-x64.rpm

Putty_v0.63.0.0.43510830.exe







This article is from the "If you bloom, the breeze came in" blog, please be sure to keep this source http://iqdutao.blog.51cto.com/2597934/1766864

The first section of Hadoop Learning: Hadoop configuration Installation

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.