Hadoop configuration (5)--Start yarn

Source: Internet
Author: User
Tags builtin hdfs dfs

Newer versions of Hadoop use the new MapReduce framework (MapReduce V2, also known as Yarn,yet another Resource negotiator).

YARN is isolated from MapReduce and is responsible for resource management and task scheduling. YARN runs on MapReduce, providing high availability and scalability.

The above-mentioned adoption./sbin/start-dfs.shstart Hadoop, just start the MapReduce environment, we can start yarn, let yarn to be responsible for resource management and task scheduling.


config mapr-site.xml file  

 
   
  
  1. mv ./etc/hadoop/mapred-site.xml.template ./etc/hadoop/mapred-site.xml

  
 
  1. <configuration>
  2.        <property>
  3.             <name>mapreduce.framework.name</name>
  4.             <value>yarn</value>
  5.        </property>
  6. </configuration>

Modifying a configuration file Yarn-site.xml :


  
 
  1. <configuration>
  2.        <property>
  3.             <name>yarn.nodemanager.aux-services</name>
  4.             <value>mapreduce_shuffle</value>
  5.            </property>
  6. </configuration>


start yarn ( ./sbin/start-dfs.sh )


 
   
  
  1. ./sbin/start-yarn.sh
Check:

After you start YARN, the method of running the instance is the same, only the resource management method, the task scheduling is different.

Observation log information can be found, when YARN is not enabled, is "mapred." Localjobrunner "After running the task, enabling YARN, is" mapred. Yarnrunner "in the running task. One of the benefits of starting YARN is that you can view the operation of a task through the Web interface:http://localhost:8088/cluster






[Email protected]:/usr/local/hadoop$./sbin/start-dfs.sh

16/06/22 20:28:42 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable

Starting namenodes on [localhost]

Localhost:starting Namenode, logging to/usr/local/hadoop/logs/hadoop-hadoop-namenode-athena.out

Localhost:starting Datanode, logging to/usr/local/hadoop/logs/hadoop-hadoop-datanode-athena.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0:starting Secondarynamenode, logging to/usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-athena.out

16/06/22 20:29:05 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable

[Email protected]:/usr/local/hadoop$./sbin/start-yarn. Sh

-SU:./sbin/start-yarn. Sh:no such file or directory

[Email protected]:/usr/local/hadoop$./sbin/start-yarn.

Start-yarn.cmd start-yarn.sh

[Email protected]:/usr/local/hadoop$./sbin/start-yarn.sh

Starting Yarn Daemons

Starting ResourceManager, logging to/usr/local/hadoop/logs/yarn-hadoop-resourcemanager-athena.out

Localhost:starting NodeManager, logging to/usr/local/hadoop/logs/yarn-hadoop-nodemanager-athena.out

[Email protected]:/usr/local/hadoop$ JPS

16896 DataNode

17395 NodeManager

17109 Secondarynamenode

16774 NameNode

17271 ResourceManager

17432 Jps

[Email protected]:/usr/local/hadoop$./bin/hdfs dfs-rm-r Output

16/06/22 20:29:46 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable

16/06/22 20:29:47 INFO fs. Trashpolicydefault:namenode Trash Configuration:deletion interval = 0 minutes, emptier interval = 0 minutes.

Deleted output

[email protected]:/usr/local/hadoop$./bin/hadoop jar./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar WordCount Input/core-site.xml Output/result




From for notes (Wiz)

Hadoop configuration (5)--Start yarn

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.