1. Role Assignment
| Ip |
Role |
Hostname |
| 192.168.18.37 |
Master/namenode/jobtracker |
HDP1 |
| 192.168.18.35 |
Slave/datanode/tasktracker |
HDP2 |
| 192.168.18.36 |
Slave/datanode/tasktracker |
HDP3 |
2. Install the JDK separatelyMkdir-p/usr/local/setup #安装JDK cd/usr/libtar-xvzf/usr/local/setup/jdk-7u75-linux-x64.tar.gz# changed to JDK7, purely personal preference mv jdk1.7.0_75 Jdk7 #增加JAVA环境变量 Vi/etc/profile at the end of the profile file, add the following line: Export Java_home=/usr/local/lib/jdk7export classpath= .: $JAVA _home/lib: $CLASSPATHexport path= $PATH: $JAVA _home/bin #修改jdk7的文件的相关权限 chown-r root:root jdk7 chmod-r 755 jdk7 #so Urce Modified profile file Source/etc/profile #测试JAVA安装 Java-versionjava version "1.7.0_75" Java (TM) SE Runtime Environment (build 1.7.0_75-B13) Java HotSpot (TM) 64-bit Server VM (build 24.75-b04, Mixed mode)
3. Modify/etc/sysconfig/network and/etc/hosts, respectively/etc/hosts this is the corresponding relationship between the specified IP and host name,/etc/sysconfig/network This is the hostname of the specified machine. /etc/hosts modified: 127.0.0.1 localhost localhost4 localhost4.localdomain4192.168.18.37 HDP1192.168.18.35 HDP2192.168.18.36 hdp3/etc/sysconfig/network Modify: Hostname= The machine name of the machine
4. Configure HDP1 SSH access HDP2 and HDP3 without password
4.1 Configuring HDP1 local password-free SSH#HDP1切到hdp用户配置key. Ssh-keygen-t Dsa-p "-F ~/.ssh/id_dsacat ~/.ssh/id_dsa.pub >>~/.ssh/authorized_keys #修改sshd_configsudo vi/etc/s Sh/sshd_config #删除 # To make the configuration of the following three lines effective rsaauthentication yespubkeyauthentication yesauthorizedkeysfile. ssh/authorized_ Keys #配置权限并重启sshd服务cd ~/.sshchmod authorized_keys CD. Chmod-r Sshsudo Service sshd restart
4.2 Configuring HDP1 to HDP2 and HDP3 without password ssh#将HDP1的authorized_keys复制到HDP2和HDP3scp. Ssh/authorized_keys HDP2:~/.SSH/AUTHORIZED_KEYS_HDP1SCP. Ssh/authorized_ Keys HDP3:~/.SSH/AUTHORIZED_KEYS_HDP1 #分别在HDP2和HDP3上将authorized_keys_hdp1加入到本地的authorized_keys中cat ~/.ssh/ AUTHORIZED_KEYS_HDP1 >> ~/.ssh/authorized_keys #测试ssh localhostssh hdp2ssh hdp3last login:thu APR 2 15:22:03 2015 From HDP1
5. Configure Hadoop files for three machinesFirst in master (HDP1) configuration, configure the configuration file to overwrite on slaves after the configuration is complete. If there is a related directory, you also need to create it in slaves. You can also copy the entire Hadoop installation directory to slaves after the configuration is complete and as the installation directory.
Add the following folder to the Hadoop installation directory:
/etc/profileExport Hadoop_prefix=/usr/local/hadoophadoop environment variables for the installation directory
etc/hadoop/hadoop-env.shExport Java_home=${java_home}export Hadoop_prefix=/usr/local/hadoopexport Hadoop_log_dir=/var/log/hadoophadoop Independent environment Variables for Deamon
etc/hadoop/yarn-env.shIndependent environment variables for export Java_home=${java_home}yarn
etc/hadoop/slaves, add slave machine name hdp2hdp3
Etc/hadoop/core-site.xml<configuration> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/ha Doop/tmp</value> <description>abase for other temporary directories.</description> </proper ty> <property> <name>fs.default.name</name> <value>hdfs://hdp1:9000</value& Gt </property> <property> <name>io.file.buffer.size</name> <value>4096</val Ue> </property></configuration>
Etc/hadoop/hdfs-site.xml<configuration> <property> & lt;name>dfs.namenode.secondary.http-address</name> <value>HDP1:9001</value> </property> <property > <name>dfs.namenode.name.dir</name> <value>file:/usr/local/hadoop/dfs/name</value> < /property> <property> <name> dfs.datanode.data.dir</name> <value>file:/usr/local/ hadoop/dfs/data</value> </property> <property> &NBSP;<NAME>DFS.REPLICATion</name> <value>2</value> </property> <property> <name>dfs.nameservices</ name> <value>hadoop-cluster1</value> </property> & nbsp <property> <name> dfs.webhdfs.enabled</name> <value>true</ value> </property></configuration>
Etc/hadoop/mapred-site.xml<configuration> <property> <name> ;mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>HDP1:10020</value> </property& Gt <property> <name>mapreduce.jobhistory.webapp.address</name> <value>hdp1:19888< ;/value> </property></configuration>
Etc/hadoop/yarn-site.xml <configuration> <property> & lt;name>yarn.nodemanager.aux-services</name> <value >mapreduce_shuffle</value> </property> < property> <name> yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property > <property> &NBSP;<NAME>YARN.R esourcemanager.address</name> <value>hdp1:8032</ value> </property> <property> &NBSp <name>yarn.resourcemanager.scheduler.address</name> <value>HDP1:8030</value> </ property> <property> <name> yarn.resourcemanager.resource-tracker.address</name> <value >HDP1:8031</value> </property> <property> & nbsp <name>yarn.resourcemanager.admin.address</name> <value>HDP1:8033</value> </property> <property> <name> yarn.resourcemanager.webapp.address</name> <value>HDP1:8088</value> </property></configuration > #将配置好的配置文件复制到Slaves我选择复制所有配置文件. Copy to the corresponding home directory before overwriting to the Hadoop installation directory to prevent permission changes. sudo scp-r/usr/local/hadoop/etc/hadoop [email protected]:~/sudo scp-r/usr/local/hadoop/etc/hadoop [email protected]:~/ #SSH到对应的Slave, then cover Etc/hadoop. I use the method of overwriting after first deleting. RM-RF/USR/LOCAL/HADOOP/ETC/HADOOP/*MV ~/hadoop/* /usr/local/hadoop/etc/hadoop/ 6. Add Hadoop environment variables make it easy to invoke commands and scripts in Hadoop/bin and hadoop/sbin without entering an absolute path each time. vi/etc/profile export path= $PATH:/usr/local/hadoop/bin:/usr/local/hadoop/sbin Source of re-source/ etc/profile
7. Start ValidationThere will be namenode and secondarynamenode processes on HDP1 after #格式化NameNode HDFs namenode-format #启动hdfs start-hdfs.sh started: [Email protected] Ro ot]$ JPS 2991 NameNode 3172 secondarynamenode 8730 JPS Slaves There will be a datanode process: [[email protected] root]$ jps2131 DataNode4 651 Jps #启动yarnstart-yarn.sh, the ResourceManager process will be increased on HDP1, and slaves will be added on NodeManager. The same can be observed with JPS.
8. Running the wordcount example
#创建一个要分析的txtvi/usr/local/hadoop/wc.txtthis is a wordcount appis a wordcount appa wordcount appwordcount Appapp #在hdfs创建相关 Catalog and upload Wc.txthdfs dfs-mkdir-p/wc/input hdfs dfs-put wc.txt/wc/input/#运行之hadoop jar Share/hadoop/mapreduce/hadoop-mapre Duce-examples-2.6.0.jar wordcount/wc/input/wc.txt/wc/output #查看结果hdfs Dfs-ls/wc/outputhdfs dfs-cat/wc/output/ part-r-00000
CentOS 6+hadoop 2.6.0 distributed cluster installation
