Hadoop queue management (specifying a queue run program)

After Hadoop was upgraded to CDH5, queue management was canceled, but the resource pool allocation was unified.

hadoop2.0 version, Hadoop uses a peer-queue organization, where administrators can divide users into several flat queues, and in each queue, one or more queue managers can be assigned to manage those users, such as killing any user's job and modifying the priority of any user job.

However, this is not enough, the need to divide resources into these queues, and follow a certain policy to complete the allocation of resources, which requires the support of the Hadoop job scheduler.

In short, in Hadoop, the organization of queues is the basis for queue management and resource allocation.

Online environment, set to three queues, respectively, is default,queue1,queue2,queue3.

When writing code, set the

Configuration conf = new Configuration ();
Conf.set ("Mapred.job.queue.name", "queue3");

The use of queue3 running procedures, fewer tasks, relatively adequate resources.

Hadoop Queue Management:

Configure queue-related information

• Configuration Properties in Mapre-site.xml

Configure information about the queue

<property>

<name>mapred.acls.enabled</name>

<value>true</value>

</property>

<property>

<name>mapred.queue.names</name>

<value>test,default</value>

</property> Property Mapred.queue.names is all the names of the queue, and in this name there must be a queue called "default"

<property>

<name>mapred.queue.my_queue.acl-submit-job</name>

<value> test</value>

</property>

<property>

<name>mapred.queue.default.acl-administer-jobs</name>

<value></value>

</property>

<property>

<name>mapred.queue.my_queue.acl-administer-jobs</name>

<value></value>

</property>

Mapred.queue.names is all the names of the queue.

• Must have a queue called "default"

mapred.queue.<queue-name>.acl-submit-job, specify which user or group can submit jobs to the queue

mapred.queue.<queue-name>.acl-administer-jobs, specify which user or group can manage all jobs in the queue.

• The amount of resources and other information that each queue has, specified in other configuration files

The mapred.queue.<queue-name>.acl-submit-job and Mapred.queue.<queue-name>.acl-administer-jobs configuration methods are as follows:

The user and user groups are separated by spaces, with "," split between users, with "," split between user groups, and if the queue has only a user group in its attribute value, a space is reserved.