hadoop2.7.3

Environment: centos6.7,hadoop2.7.3, Virtual machine vmware

1. Download hadoop:http://apache.fayea.com/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz

2. Namendoe 192.168.137.9; Secondnode 192.168.137.15; Datanode 192.168.137.16

3. Modify the/etc/hosts of three hosts and add the Namenode,secondnode,datanode information separately

[Email protected] ~]# cat/etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.137.9 namenode192.168.137.15 secondnode192.168.137.16 Datanode

4. Download JDK:jdk-8u77-linux-x64.tar.gz

5. Installing Java

①yum Remove Java-y

②tar zxvf jdk-8u77-linux-x64.tar.gz

③MV Jdk1.8.0_77/usr/local/java

④vi/etc/profile

Export java_home=/usr/local/javaexportclasspath=.: $JAVA _home/jre/lib/rt.jar: $JAVA _home/lib/dt.jar: $JAVA _home/ Lib/tools.jarexport path= $PATH: $JAVA _home/bin

⑤source/etc/profile

[[email protected] src]# Java-versionjava version "1.8.0_77" Java (TM) SE Runtime Environment (build 1.8.0_77-b03) Java Hots Pot (TM) 64-bit Server VM (build 25.77-b03, Mixed mode)

Three hosts do the above command operation.

6. Environment variable Optimization:

Cat << EOF > ~/.toprcrcfile for "Top withwindows" # shameless braggin ' id:a, mode_altscr=0, Mode_irixps=1,delay_ time=3.000, Curwin=0def fieldscur=aehioqtwknmbcdfgjplrsuvyzx winflags=32569, sortindx=10, maxtasks=0 SUMMC Lr=1, Msgsclr=1, Headclr=3,taskclr=2job fieldscur=abcefgjlrstuvyzmknhiwopqdxwinflags=62777, sortindx=0, maxtasks= 0summclr=6, Msgsclr=6, headclr=7, Taskclr=6mem fieldscur=anopqrstuvbcdefgjlmyzwhikxwinflags=62777, sortindx=13, Maxtasks=0summclr=5, Msgsclr=5, headclr=4, Taskclr=5usr fieldscur=abdecgfhijlopqrstuvyzmknwxwinflags=62777, Sortindx=4, Maxtasks=0summclr=3, msgsclr=3, headclr=2, taskclr=3eof

Continue environment variable Optimization:

Vim/etc/security/limits.conf

Hadoop-nofile 32768hadoop-nproc 32000

Continue environment variable Optimization:

Vim/etc/pam.d/system-auth

Auth Required pam_limits.so

All node operations.

7. Create a Hadoop user

Useradd-u the Hadoop && echo "Hadoop" |passwd--stdin Hadoop

Mkdir/data &&chown-r Hadoop.hadoop/data

all node operations .

8. Password-Free Login

①su-hadoop

②ssh-keygen

③ on the Namenode:

VI. Ssh/authorized_keys

Add the. Ssh/id_rsa.pub content of all nodes and distribute them to each node.

chmod. Ssh/authorized_keys

9.namenode Operation:

Unzip Hadoop,

Tar zxvf hadoop-2.7.3.tar.gz

To move a directory:

MV hadoop-2.7.3/home/hadoop/hadoop2.7.3

10. Operation per node:

Vim/home/hadoop/.bash_profile

Modify:

Modified: Export Hadoop_home=/home/hadoop/hadoop2.7.3export path= $PATH: $HADOOP _home: $HADOOP _home/bin: $HADOOP _home/ Sbinexport Hadoop_home_warn_suppress=1export PATH

$source/home/hadoop/.bash_profile

Operation on 11.namenode:

$CD/home/hadoop/hadoop2.7.3/etc/hadoop

$vim hadoop-env.sh

Modify:

Export Java_home=/usr/local/java

Increase:

Export Hadoop_prefix=/home/hadoop/hadoop2.7.3export hadoop_heapsize=15000

$vim yarn-env.sh

Modify:

Export Java_home=/usr/local/java

$vim mapred-env.sh

Modify:

Export Java_home=/usr/local/java

$ VI Hdfs-site.xml

<?xml version= "1.0"  encoding= "UTF-8"? ><?xml-stylesheet type= "Text/xsl"  href= " Configuration.xsl "?><configuration>  <property>         <name>dfs.namenode.http-address</name>         <value>namenode:50070</value>        <description>  NameNode  through current parameters   get  fsimage  and  edits </description>  </ Property>  <property>        <name> Dfs.namenode.secondary.http-address</name>        <value> secondnode:50090</value>        <description>  secondnamenode  with current parameters   get the latest  fsimage </description>    </ Property>    <property>        <name>dfs.replication</name>         <value>2</value>         <description>  set  HDFS  Save copy number of files, default is 3 </description>     </property>   <property>        < Name>dfs.namenode.checkpoint.dir</name>        <value> file:///home/hadoop/hadoop2.7.3/hdfs/namesecondary</value>         <description>  Settings  secondary  storage   Temporary image   Local file system path, if this is a comma-delimited list of files, The image will be copied redundantly to all directories, only for  secondary  effective  </description>    </property>     <property>        <name> Dfs.webhdfs.enabled</name>        <value>true</value>    </property>     <property>        <name> dfs.namenode.name.dir</name>        <value>file:///data/ Work/hdfs/name/</value>        <description> namenode   Local file system paths for persisting namespaces and exchanging logs  </description>    </property>     <property>        <name>dfs.datanode.data.dir </name>        <value>file:///data/work/hdfs</value>         <description> DataNode  a directory listing of local block files, separated by commas  </description>    </property>    <property>         <name>dfs.stream-buffer-size</name>        < value>131072</value>        <description>  default is 4KB, As a Hadoop buffer, for Hadoop to read HDFs files and write HDFs files, as well as map of the output of this buffer capacity, for the current hardware is very conservative, can be set to 128k (131072), or even 1M ( It's too big. Map and reduce tasks may overflow memory)  </description>    </property>     <property>        <name> dfs.namenode.checkpoint.period</name>        <value>3600 </value>        <description>  two times  checkpoints The interval between  , in seconds, only for  secondary  effective  </description>    </property> </configuration>

Specific information can be viewed on the official website: http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

$vim Mapred-site.xml

<?xml version= "1.0"? ><?xml-stylesheet type= "text/xsl" href= "configuration.xsl"?><configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value></property ></configuration>

$vim Yarn-site.xml

Modify:

<?xml version= "1.0"?><configuration><property> <name>yarn.nodemanager.aux-services</ Name> <value>mapreduce_shuffle</value></property></configuration>

$ VI Core-site.xml

Modify:

<?xml version= "1.0" encoding= "UTF-8"? ><?xml-stylesheet type= "text/xsl" href= "configuration.xsl"?>< Configuration><property> <name>fs.defaultFS</name> <value>hdfs://namenode:9000/</ value> <description> set Namenode hostname and port </description></property><property> <name> ;hadoop.tmp.dir</name> <value>/home/hadoop/tmp</value> <description> The directory where temporary files are stored </ Description></property></configuration>

For specific reference: Http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-common/core-default.xml

New directory on all nodes

$mkdir/home/hadoop/tmp$mkdir/data/work/hdfs/namesecondary-p

13.namenode on

$start-all.sh

This article is from the "Echo Xiayun" blog, so be sure to keep this source http://linuxerxy.blog.51cto.com/10707334/1877842

hadoop2.7.3