Hadoop1
- Configure Mapred-site.xml to add the following:
<property > <name>Mapred.jobtracker.taskScheduler</name> <value>Org.apache.hadoop.mapred.FairScheduler</value> </Property > <property > <name>Mapred.fairscheduler.allocation.file</name> <value>/etc/hadoop/conf/pools.xml</value> </Property >
- Configure Pools.xml to add the following:
<queue name= "Default" > <minResources>1024x768Mb,1vcores</minresources><maxResources>61440Mb,20vcores</maxresources><maxRunningApps>Ten</maxRunningApps><weight>2.0</weight><schedulingPolicy>Fair</schedulingPolicy></queue><queue name= "Hadoop" > <minResources>1024x768Mb,10vcores</minresources><maxResources>3072000Mb,960vcores</maxresources><maxRunningApps> -</maxRunningApps><weight>5.0</weight><schedulingPolicy>Fair</schedulingPolicy><aclSubmitApps>Hadoop,yarn,spark</aclsubmitapps></queue><queue name= "Spark" > <minResources>1024x768Mb,10vcores</minresources><maxResources>61440Mb,20vcores</maxresources><maxRunningApps>Ten</maxRunningApps><weight>4.0</weight><schedulingPolicy>Fair</schedulingPolicy><aclSubmitApps>Yarn,spark</aclsubmitapps></queue><userMaxAppsDefault> -</userMaxAppsDefault>
- Submit Job Specify queue mode
-Dmapred.job.queue.name=hadoop
Hadoop2
- Configure Yarn-site.xml to add the following:
<property> <name>yarn. ResourceManager. Scheduler. Class</name> <value>org. Apache. Hadoop. Yarn. Server. ResourceManager. Scheduler. Fair. Fairscheduler</value></property><property> <name>yarn. Scheduler. Fair. Allocation. File</name> <value>/home/cluster/conf/hadoop/fair-scheduler. XML</value></property><property> <name>yarn. Scheduler. Fair. User-as-default-queue</name>//If you want to use a user name as a queue, you can configure this property to True, which defaults to true, so if you do not want the user name as a queue, you must explicitly set it to false <value> False</value> </property>
- Configure Fair-scheduler.xml to add the following:
<queue name= "Default" > <minResources>1024x768Mb,1vcores</minresources><maxResources>61440Mb,20vcores</maxresources><maxRunningApps>Ten</maxRunningApps><weight>2.0</weight><schedulingPolicy>Fair</schedulingPolicy></queue><queue name= "Hadoop" > <minResources>1024x768Mb,10vcores</minresources><maxResources>3072000Mb,960vcores</maxresources><maxRunningApps> -</maxRunningApps><weight>5.0</weight><schedulingPolicy>Fair</schedulingPolicy><aclSubmitApps>Hadoop,yarn,spark</aclsubmitapps></queue><queue name= "Spark" > <minResources>1024x768Mb,10vcores</minresources><maxResources>61440Mb,20vcores</maxresources><maxRunningApps>Ten</maxRunningApps><weight>4.0</weight><schedulingPolicy>Fair</schedulingPolicy><aclSubmitApps>Yarn,spark</aclsubmitapps></queue><userMaxAppsDefault> -</userMaxAppsDefault>
- Submit Job Specify queue mode
-Dmapreduce.job.queuename=root.hadoop
Spark
- Submit Job Specify queue mode
--queue=root.spark
Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.
HADOOP1 & HADOOP2 Fair-schduler configuration and use