Hadoop Yarn (i)--single-machine pseudo-distributed environment installation

Careers:

Our name is LAN.

Our core technical team from Ali, Huawei, Kingdee, Mobile, GE, etc.

We obtained the investment from Alibaba Group co-founder, Lakeside Shannan President Shei, IDG partner Newqui, follow, etc.

Our website: https://www.dtwave.com

We offer: options, five risks, full probationary period, commercial insurance, free physical examination, entry, weekly team building, birthday party, birthday gift, holiday gift, technology sharing, technical topic group, etc.

Our site is in front of Hangzhou Yuhang, the dream Town and Beijing Wangjing Soho

We are a strong technical atmosphere, high executive power, full of passion of the team, look forward to your joining, and grow together. If you are interested, please send your CV, we will quickly deal with:

Data development engineers, node.js, algorithmic engineers, Test development engineers, architects, please deliver your CV to qin.xq@dtwave-inc.com

Java engineer, front-end engineer, please send your CV to min.wm@dtwave-inc.com

Operational engineer, Project Manager, please send your CV to yurong.lyr@dtwave-inc.com

Hamawhite (qq:530422429) original works, reproduced please indicate the source: http://write.blog.csdn.net/postedit/40556267.

This article is based on the Hadoop website installation tutorial written by Hadoop yarn in a stand-alone pseudo distributed environment of the installation report, for reference only.
1. The installation environment is as follows:
System: Ubuntu14.04
Hadoop version: hadoop-2.5.0
Java version: openjdk-1.7.0_55
2. Download Hadoop-2.5.0, http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.5.0/hadoop-2.5.0.tar.gz
of this paper$HADOOP _homeFor:/home/baisong/hadoop-2.5.0(user name is Baisong).
Add environment variables to the ~/.BASHRC file as follows:
Export hadoop_home=/home/baisong/hadoop-2.5.0
Then compile the command as follows:
$ source ~/.BASHRC
3. Install JDK and set java_home environment variable. At the end of the/etc/profile file, add the following
Export java_home=/usr/lib/jvm/java-7-openjdk-i386//depending on your JAVA installation directory
Export path= $JAVA _home/bin: $PATH
Enter the following command to take the configuration into effect
$ source/etc/profile
4. Configure SSH. First generate the secret key, the command is as follows, and then all the way return to confirm, do not need any input.
$ ssh-keygen-t RSA then writes the public key to the Authorized_keys file with the following command:
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Finally, enter the following command, as promptedYesCan.
$ ssh localhost
5. Modify the HADOOP configuration file and enter the ${hadoop_home}/etc/hadoop/directory.
1 Set the environment variable, add the Java installation directory in hadoop-env.sh, as follows:
Export java_home=/usr/lib/jvm/java-7-openjdk-i386
2) Modify Core-site.xml, add the following content.
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/baisong/hadooptmp</value>
</property>
Note: Hadoop.tmp.dir items are optional (the above settings need to be manually created Hadooptmp folders).
3) Modify the Hdfs-site.xml and add the following contents ".
<property>
<name>dfs.repliacation</name>
<value>1</value>
</property>
4 Rename the Mapred-site.xml.template to Mapred-site.xml and add the following content.
$ mv Mapred-site.xml.template mapred-site.xml//renaming
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
5) Modify Yarn-site.xml and add the following content.
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
6. Format HDFs, command as follows:
Bin/hdfs Namenode-format Note: Bin/hadoop namenode-format command is obsolete
A successful format will create a Dfs folder in/home/baisong/hadooptmp.
7. Start HDFs, order as follows:
$ sbin/start-dfs.sh
Encountered the following error:

14/10/29 16:49:01 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable St Arting namenodes on [OpenJDK Server VM warning:you have loaded LIBRARY/HOME/BAISONG/HADOOP-2.5.0/LIB/NATIVE/LIBHADOOP.S o.1.0.0 which might have disabled stack guard.
The VM would try to fix the stack guard now.
It's highly recommended that's the fix the library with ' execstack-c <libfile> ', or link it with '-Z noexecstack '. localhost] sed:-E expression #1, char 6:unknown option to ' s ' VM:ssh:Could not resolve hostname-vm:name or service No T known library:ssh:Could not resolve hostname library:name or service not known have:ssh:Could not resolve hostname Have:name or service not known which:ssh:Could not resolve hostname which:name or service not known Might:ssh:Could Not resolve hostname might:name or service not known warning:: Ssh:could not resolve hostname:: Name or service Not known Loaded:ssh:CouldNot resolve hostname loaded:name or service not known have:ssh:Could not resolve hostname or service not have:name WN Server:ssh:Could not resolve hostname server:name or service not known

For analysis reasons, the Hadoop_common_lib_native_dir and hadoop_opts environment variables are not set, and the following content is added to the ~/.BASHRC file and compiled.
Export hadoop_common_lib_native_dir= $HADOOP _home/lib/native
Export hadoop_opts= "-djava.library.path= $HADOOP _home/lib"
$ source ~/.BASHRC
Restart HDFs, the output is as follows, indicating successful startup.

You can use the Web interface to view Namen ODE Health, the URL is http://localhost:50070
The command to stop HDFs is:
$ sbin/stop-dfs.sh
8. Start yarn, order as follows:
$ sbin/start-yarn.sh
You can use the Web interface to view Namenode health, the URL is http://localhost:8088
The command to stop HDFs is:
$ sbin/stop-yarn.sh
After the above HDFs and yarn startup is complete, you can see whether the startup succeeds through the JPS command.

9. Run the test program.
1 The test calculates PI, the command is as follows:
$ bin/hadoop jar Share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar Pi 20 10
2 test grep, first need to upload the input file to the HDFs, the command is as follows:
$ Bin/hdfs dfs-put Etc/hadoop input
Run the GREP program with the following commands:
$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar grep input Output ' dfs[a-z.] +'
The output of the run results is as follows:
10. Add environment variables to facilitate the use of start-dfs.sh, start-yarn.sh and other commands (optional).
Add environment variables to the ~/.BASHRC file as follows:
Export path= $HADOOP _home/bin: $HADOOP _home/sbin: $PATH
Then compile the command as follows:
$ source ~/.BASHRC
The following figure is a variable added to the ~/.BASHRC file for reference.