配置hadoop 使用fair scheduler調度器

hadoop版本為cloudera hadoop cdh3u3

配置步驟為

1.  將$HADOOP_HOME/contrib/fairscheduler/hadoop-fairscheduler-0.20.2-cdh3u3.jar拷貝到$HADOOP_HOME/lib檔案夾中

2.  修改$HADOOP_HOME/conf/mapred-site.xml設定檔

 <property>    <name>mapred.jobtracker.taskScheduler</name>    <value>org.apache.hadoop.mapred.FairScheduler</value>  </property>  <property>      <name>mapred.fairscheduler.allocation.file</name>      <value>/home/hadoop/hadoop-0.20.2-cdh3u3/conf/fair-scheduler.xml</value>   </property>   <property>     <name>mapred.fairscheduler.preemption</name>     <value>true</value>   </property>   <property>     <name>mapred.fairscheduler.assignmultiple</name>     <value>true</value>   </property>   <property>     <name>mapred.fairscheduler.poolnameproperty</name>     <value>mapred.queue.name</value>     <description>job.set("mapred.queue.name",pool); // pool is set to either 'high' or 'low' </description>   </property>   <property>     <name>mapred.fairscheduler.preemption.only.log</name>     <value>true</value>   </property>   <property>     <name>mapred.fairscheduler.preemption.interval</name>     <value>15000</value>   </property>      <property>     <name>mapred.queue.names</name>     <value>default,hadoop,hive</value>   </property>

3.  在$HADOOP_HOME/conf/建立設定檔fair-scheduler.xml

<?xml version="1.0"?><allocations><pool name="hive">  <minMaps>90</minMaps>  <minReduces>20</minReduces>  <maxRunningJobs>20</maxRunningJobs>  <weight>2.0</weight>  <minSharePreemptionTimeout>30</minSharePreemptionTimeout></pool><pool name="hadoop">  <minMaps>9</minMaps>  <minReduces>2</minReduces>  <maxRunningJobs>20</maxRunningJobs>  <weight>1.0</weight>  <minSharePreemptionTimeout>30</minSharePreemptionTimeout></pool><user name="hadoop">    <maxRunningJobs>6</maxRunningJobs></user><poolMaxJobsDefault>10</poolMaxJobsDefault><userMaxJobsDefault>8</userMaxJobsDefault><defaultMinSharePreemptionTimeout>600</defaultMinSharePreemptionTimeout><fairSharePreemptionTimeout>600</fairSharePreemptionTimeout></allocations>

4.  在叢集的各個節點執行以上步驟，然後重啟叢集，在http://namenode:50030/scheduler  即可查看到調度器運行狀態，如果修改調度器配置的話，只需要修改檔案fair-scheduler.xml ，不需重啟配置即可生效。

5. 在執行hive任務時，設定hive屬於的隊列set mapred.job.queue.name=hive;

##########

另外，如果在執行MR JOB的時候出現XX使用者訪問不了YY隊列的話，就需要在mapred-queue-acls.xml裡配置相應的屬性，來對存取權限進行控制，比如：

<property>  <name>mapred.queue.default.acl-submit-job</name>  <value>*</value>  <description> Comma separated list of user and group names that are allowed    to submit jobs to the 'default' queue. The user list and the group list    are separated by a blank. For e.g. user1,user2 group1,group2.    If set to the special value '*', it means all users are allowed to    submit jobs. If set to ' '(i.e. space), no user will be allowed to submit    jobs.    It is only used if authorization is enabled in Map/Reduce by setting the    configuration property mapred.acls.enabled to true.    Irrespective of this ACL configuration, the user who started the cluster and    cluster administrators configured via    mapreduce.cluster.administrators can submit jobs.  </description></property><property>  <name>mapred.queue.default.acl-administer-jobs</name>  <value>*</value>  <description> Comma separated list of user and group names that are allowed    to view job details, kill jobs or modify job's priority for all the jobs    in the 'default' queue. The user list and the group list    are separated by a blank. For e.g. user1,user2 group1,group2.    If set to the special value '*', it means all users are allowed to do    this operation. If set to ' '(i.e. space), no user will be allowed to do    this operation.    It is only used if authorization is enabled in Map/Reduce by setting the    configuration property mapred.acls.enabled to true.    Irrespective of this ACL configuration, the user who started the cluster and    cluster administrators configured via    mapreduce.cluster.administrators can do the above operations on all the jobs    in all the queues. The job owner can do all the above operations on his/her    job irrespective of this ACL configuration.  </description></property>