Hadoop Learning (i) Hadoop pseudo-distributed environment building

Pre-Preparation 1. Create a Hadoop-related directory (easy to manage) 2, give Hadoop users and all group permissions to the/opt/* directory

sudo chrown-r hadoop:hadoop/opt/*

3, JDK installation and configuration configuration Hdfs/yarn/mamreduce1, decompression Hadoop

TAR-ZXF hadoop-2.5.0.tar.gz-c/opt/modules/(delete Doc's help document, save space) rm-rf/opt/modules/hadoop-2.5.0/share/doc/

2, Configuration hadoop-env.sh

CD/OPT/MODULES/HADOOP-2.5.0/ETC/HADOOPVI Hadoop-env.shexport java_home=/usr/local/jdk1.7.0_67 Verify that the configuration is successful/opt/ Modules/hadoop-2.5.0/bin/hadoop

3, Configuration Core-site.xml

<!--Specify the location of the machine where the Namenode is located and access the interactive port number--><property> <name>fs.defaultFS</name> <value>hdfs:/ /hadoop.lianwei.org (hostname):8020</value></property><!--Specify a temporary directory for the Hadoop runtime--><property> < Name>hadoop.tmp.dir</name> <value>/opt/modules/hadoop-2.5.0/data/tmp</value></property >

4, Configuration Slaves

Vi/opt/module/hadoop-2.5.0/etc/hadoop/slaves change the content to the specified Datanode machine location: hadoop.lianwei.org

5, Configuration Hdfs-site.xml

vi/opt/module/hadoop-2.5.0/etc/hadoop/hdfs-site.xml<!--Specifies the number of copies of a block of files in the system--><property> <name> Dfs.replication</name> <value>1</value> (default is 3 in distributed) </property>

6. Formatted Namenode

Bin/hdfs Namenode-format Note: dfs/name/current/fsimage_* (image file) will be generated under Hadoop.tmp.dir

7. Start HDFs

sbin/hadoop-daemon.sh start namenodesbin/hadoop-daemon.sh start Datanodejps (see if it started successfully)

8. Browser View Web Console

http://hadoop.lianwei.org:50070

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "alt=" image "Style=" Background:url ("/e/u261/ Lang/zh-cn/images/localimage.png ") no-repeat center;border:1px solid #ddd;"/>

9, Configuration Yarn-site.xml

<property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value ></property> Specify the location of the machine where the Resorcemanager is located:<property> <name>yarn.resourcemanager.hostname</ Name> <value>hadoop.lianwei.org</value></property>

10, start ResourceManager, NodeManager

$ sbin/yarn-daemon.sh Start resourcemanager$ sbin/yarn-daemon.sh start NodeManager

11. View yarn Web-ui interface via browser

http://hadoop.lianwei.org:8088

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "alt=" image "Style=" Background:url ("/e/u261/ Lang/zh-cn/images/localimage.png ") no-repeat center;border:1px solid #ddd;"/>

12, Configuration Mapred-site.xml

Specify mapreduce to run on yarn <property> <name>mapreduce.framework.name</name> <value>yarn</ Value></property>

13. Test WordCount Case Run

Create a pending file directory in HDFs bin/hdfs dfs-mkdir-p/user/lianwei/mapreduce/examples/wordcount/input upload pending files to the pending directory in HDFs Bin/hdfs Dfs-put/opt/data/lv.input/user/lianwei/mapreduce/examples/wordcount/input execute command bin/yarn jar Share/hadoop/mapreduce /hadoop-mapreduce-examples-2.5.0.jar wordcount/user/lianwei/mapreduce/examples/wordcount/input/user/lianwei/ Mapreduce/examples/wordcount/output (output directory cannot be created manually and cannot exist, otherwise it will overwrite)

14. View the results of the run in the Web UI

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "alt=" image "Style=" Background:url ("/e/u261/ Lang/zh-cn/images/localimage.png ") no-repeat center;border:1px solid #ddd;"/>

Precautions:

1, prior to the CentOS version of the firewall is iptables, but centos7.0 default is to use firewall as a firewall systemctl stop Firewalld.service #停止firewallsystemctl Disable Firewalld.service #禁止firewall开机启动firewall-cmd--state #查看默认防火墙状态 (show notrunning after turn off, show running when turned on)

This article is from the "11954530" blog, please be sure to keep this source http://11964530.blog.51cto.com/11954530/1837650

Hadoop Learning (i) Hadoop pseudo-distributed environment building