Hadoop queue management (specifying a queue run program)

Source: Internet
Author: User

After Hadoop was upgraded to CDH5, queue management was canceled, but the resource pool allocation was unified.





hadoop2.0 version, Hadoop uses a peer-queue organization, where administrators can divide users into several flat queues, and in each queue, one or more queue managers can be assigned to manage those users, such as killing any user's job and modifying the priority of any user job.

However, this is not enough, the need to divide resources into these queues, and follow a certain policy to complete the allocation of resources, which requires the support of the Hadoop job scheduler.

In short, in Hadoop, the organization of queues is the basis for queue management and resource allocation.


Online environment, set to three queues, respectively, is default,queue1,queue2,queue3.

When writing code, set the

Configuration conf = new Configuration ();
Conf.set ("Mapred.job.queue.name", "queue3");

The use of queue3 running procedures, fewer tasks, relatively adequate resources.



Hadoop Queue Management:

Configure queue-related information

• Configuration Properties in Mapre-site.xml

Configure information about the queue

<property>

<name>mapred.acls.enabled</name>

<value>true</value>

</property>

<property>

<name>mapred.queue.names</name>

<value>test,default</value>

</property> Property Mapred.queue.names is all the names of the queue, and in this name there must be a queue called "default"

<property>

<name>mapred.queue.my_queue.acl-submit-job</name>

<value> test</value>

</property>

<property>

<name>mapred.queue.default.acl-administer-jobs</name>

<value></value>

</property>

<property>

<name>mapred.queue.my_queue.acl-administer-jobs</name>

<value></value>

</property>

Mapred.queue.names is all the names of the queue.

• Must have a queue called "default"

mapred.queue.<queue-name>.acl-submit-job, specify which user or group can submit jobs to the queue

mapred.queue.<queue-name>.acl-administer-jobs, specify which user or group can manage all jobs in the queue.

• The amount of resources and other information that each queue has, specified in other configuration files

The mapred.queue.<queue-name>.acl-submit-job and Mapred.queue.<queue-name>.acl-administer-jobs configuration methods are as follows:

The user and user groups are separated by spaces, with "," split between users, with "," split between user groups, and if the queue has only a user group in its attribute value, a space is reserved.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.