Hadoop yarn Installation

Hadoop yarn has solved many of the problems in MRv1, installing a Hadoop yarn, and then easy to learn Spark,yarn

Issues such as/etc/hosts,ssh password login in the first edition of Hadoop are not detailed here, but this is just a little bit about the basic configuration of yarn and Hadoop version1.

The basic three profiles will be different, listing my own three files configuration, but also the most basic configuration, and then give the Yarn-site.xml configuration, and finally solve a problem encountered.

1,core-site.xml file

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs:// 10.11.1.42:9100</value>
        </property>
</configuration>

2,map-site.xml file

<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value >yarn</value>
        </property>
</configuration>

3,hdfs-site.xml file

<configuration>
        <property>
                <name>dfs.replication</name>
                <value>2< /value>
        </property>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>/home/egraldlo/hadoop/yarn/hadoop_related/data1,/home/egraldlo/hadoop/yarn/hadoop_related/data2 </value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>/home/egraldlo/hadoop/yarn/hadoop_related/data</value>
        </property>
</configuration>

4,yarn-site.xml file

<configuration> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle .class</name> <value>mapreduce.shuffle</value> </property> <pro Perty> <name>yarn.resourcemanager.address</name> <value>10.11.1.42:18 040</value> </property> <property> <name>yarn.resourcemanager.sche Duler.address</name> <value>10.11.1.42:18030</value> </property> & Lt;property> <name>yarn.resourcemanager.webapp.address</name> <value> 10.11.1.42:18088</value> </property> <property> <name>yarn.reso Urcemanager.resource-tracker.address</name> <value>10.11.1.42:8025</value> < /property> </confiGuration>


 

The problems encountered:

FATAL org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:Failed to initialize Mapreduce.shuffle Java.lang.IllegalArgumentException:The ServiceName:mapreduce.shuffle Set in Yarn.nodemanager.aux-services is invalid. The valid service name should only contain a-za-z0-9_ and can don't start with numbers

Workaround:

The mapreduce.shuffle in the Yarn-site.xml file is changed to Mapreduce_shuffle

        <property>
                <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
                < Value>mapreduce_shuffle</value>
        </property>