A series of thoughts raised by "How to run a hive SQL statement using the specified queue (Yarn) After Beeline connection HiveServer2"

background  The version of HiveServer2 we use is 0.13.1-cdh5.3.2, and the current tasks are built using hive SQL in two types: manual tasks (ad hoc analysis requirements), scheduling tasks (general analysis requirements), both submitted through our web system. The previous two types of tasks were submitted to a queue called "Hive" in yarn, in order to prevent the two types of tasks from being affected and the number of parallel tasks causing the "hive" queue resource to be strained, we built a task buffer queue in the dispatch system, all the submitted Tasks (manual task , scheduling tasks) are not directly committed to the cluster, but are committed to the buffer queue, and then a fixed number of worker threads then commit the buffer queue's task out to the cluster execution.   This mode of operation is not a lot of running the task is still feasible, once our regular analysis requirements become more, the number of scheduled tasks will increase, then the temporary analysis needs to establish a manual task may affect the execution of scheduling tasks, the impact of the main performance in the following two aspects:  (1) The manual task and the scheduling task share a queue "hive", while the queue "hive" resource allocation is mainly based on the resource consumption of scheduling tasks allocated, so that the random submission of manual tasks and scheduling tasks together, it may affect the execution speed of the original scheduling task And it is not possible to choose the appropriate scheduling strategy based on the respective characteristics of the two tasks (e.g. FIFO, Fair);  (2) manual task submission needs to go through the buffer queue inside the dispatch system, if the scheduling task is more, The manual task may need to wait a long time in the buffer queue to commit to the cluster to run, while the resource required to run the task needs to be shared with the dispatch task, the execution time is not guaranteed, the user usually waits for a long time;  for the above situation, we have the queue "hive" To split: Hive.temporary, Hive.common, where hive.temporary is used for the temporary analysis of the requirements of the manual task, Hive.common for the general analysis of the requirements of the scheduling task, so that we can according to the actual situation of the business, the two flexible resource allocation.   After the queue is split, we have the following thoughts: Do manual tasks (temporary analysis requirements) still require a buffer queue of the dispatch system to commit to the cluster?   Queue splitting to some extent means that the resource is independent, that is, the manual task and the scheduling task to achieve physical isolation, if the manual task is still using the dispatch system's buffer queue, it must also be dedicated to the manual task queue, and the manual task is temporary analysis requirements, it is not possible to reserve too many resources to it , in order to ensure that the manual task executes within an acceptable time, we must restrict the queue "Hive.teMporary "can run the number of tasks at the same time to ensure that the running task has sufficient resources to execute, and if a manual task submitted to the cluster at a point in time exceeds the queue" hive.temporary "to run the limit of the number of tasks concurrently, they will be temporarily" stacked " In the queue of yarn scheduler.   Thus, we do not need to set up a dedicated buffer queue for manual tasks in the dispatch system, directly using the Yarn scheduler queue. We can directly submit the manual task directly to the cluster, the yarn scheduler according to the queue "hive.temporary" according to the scheduling policy and the current queue resource usage to determine whether to run the task.   Because current manual tasks are submitted directly to the cluster without a buffer queue of the dispatch system, the submission of manual tasks can no longer be limited to the web system, and users can submit hive SQL directly via the Beeline client, which is officially recommended by Hive.   This involves the core issue of this article: How do I run a hive sql statement using the specified queue (Yarn) After Beeline connection HiveServer2?   Solution The designation of the  mapreduce run queue is specified by configuring the (Configuration) property "Mapreduce.job.queuename".   The first thing you might think of is the "set Mapreduce.job.queuename=queuename" way to pick the run queue, which is undesirable in the scenario of a manual task (temporary analysis requirement), as mentioned earlier, We specifically assigned the corresponding queue resource "Hive.temporary" for this similar task, and we must be able to ensure that the user submits the hive SQL statement through Beeline Connection HiveServer2 and runs in the specified queue "Hive.temporary". And the user cannot change the run queue arbitrarily, that is, the property "Mapreduce.job.queuename" cannot be changed arbitrarily.   The rights control policies currently used by HiveServer2 are sql standard based hive authorization and storage based  authorization in the metastore server. Where Sql standard based hive authorization will have the Hive Terminal command "set" Limitations: Only the attributes listed in the whitelist (hive.security.authorization.sqlstd.confwhitelist) can be assigned. The whitelist contains a batch of attributes, including "Mapreduce.job.queuename", which we need to hive-site.xml by the parameter "--hiveconf" when we start the HiveServer2 or by setting the white list " Hive.security.authorization.sqlstd.confwhitelist "value, excluding the attribute" Mapreduce.job.queuename ", It is not possible for our users to change the value of "mapreduce.job.queuename" after connecting to HiveServer2 via Beeline.   Since the user cannot change the property "Mapreduce.job.queuename", the property "Mapreduce.job.queuename" must have a default value after HiveServer2 startup, i.e. "HIVE.temporary "so that the hive SQL submitted by the user after the Beeline connection HiveServer2 is run in the queue" Hive.temporary ". So, the next question is, what if this default setting is completed?   In general, we would think that the operation of HIVESERVER2 involves at least two configuration files:  (1) hadoop:core-site.xml, Hdfs-site.xml, Mapred-site.xml, yarn-site.xml  (2) hive:hive-site.xml  the attribute values in these configuration files are "packaged" into the configuration properties of the MapReduce task. We naturally think of specifying "Mapreduce.job.queuename" in the Mapred-site.xml or hive-site.xml, but the actual verification shows that this is not the case.   (1) specify "Mapreduce.job.queuename";   (2) in Hive-site.xml (mapred-site.xml) to test whether the specified value "test" is in effect;    We found that the value of the attribute "Mapreduce.job.queuename" was replaced with "Root.hdfs" after the account "HDFs" was connected to HiveServer2 by Beeline, and the actual execution of hive SQL statement, the corresponding MapReduce task is also found to be committed to the queue "Root.hdfs".   also has a question for this type of situation on the Internet: http://mail-archives.apache.org/mod_mbox/hive-user/201410.mbox/%[email protected] .com%3e   One solution is as follows:   the essence of this is to pass the property value by the URL parameter when the connection is established.   But this approach is not useful for us, and we can't force our users to specify queues when connecting to HiveServer2, but this solution gives us a thought: since we can initialize the property value when the client establishes a connection, the server Will there be an initialization operation when I accept a connection?   Through the source code and the log output of various "DEBUG", and finally found the problem. As we all know, HiveServer2 has a "session" concept, that is, each user has his own session ("Set" function is to set a specific property value in the user's own session), the user session is isolated from each other. According to our guess, we found HiveServer2. Session implementation class: Org.apache.hive.service.cli.session.HiveSessionImpl.HiveSessionImpl, its initialization (constructor) is where "magic" occurs, the key fragment is as follows:    and the Confvars.hive_server2_map_fair_scheduler_queue value defaults to True, as follows:   the code snippet above is bound to be executed by default. This means that Hivesession's build process involves the specific use of Hadoop scheduler. The function of  refreshdefaultqueue is to select a queue for a specific user according to a fair scheduling policy, where available:  (1) MapReduce2; (2) The user name is not NULL, and (3) the use of a fair scheduling strategy;    continue to follow the code, you can see the following code snippet: ... the role of   queueplacementpolicy Assignapptoqueue is to assign queues to users, Where the value of Yarnconfiguration.default_queue_name is Default,mr2_job_queue_ The value of the property is Mapreduce.job.queuename,conf is actually a hiveconf instance, and the value of mapreduce.job.queuename that we specify in Hive-site.xml or Mapred-site.xml is here Overwritten (note: in each user sessionKeep their own conf).    from the above code can be seen, ultimately by a certain queueplacementrule to determine the user queue, then what determines rules?    where Fairschedulerconfiguration.user_as_default_ The value of the queue is Yarn.scheduler.fair.user-as-default-queue,fairschedulerconfiguration.default_user_as_default_ The value of the queue is true, which means that the rules contains two Queueplacementrule instances by default:  (1) org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.queueplacementrule.specified   as mentioned above, The Requestedqueue value is "default", so the rule returns a value of "" (an empty string) and continues the selection of the next rule's queue.   (2) org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.queueplacementrule.user   The rule queue selection rules are particularly simple: root. + user name.   This is why the queue "Root.hdfs" appears in the verification process above.   ConclusionIn summary, the user through the Beeline connection HiveServer2 after the queue selection, by default, the impact of the fair scheduling policy, If you want to specify mapreduce.job.queuename through Hive-site.xml or mapred-site.xml, a very simple way is to hive.server2.map.fair.scheduler.queue the property value (CONFV Ars. Hive_server2_map_fair_scheduler_queue) to False, Can be specified in Hive-site.xml or by parameter when HiveServer2 is started, so the selection of HiveServer2 queues is no longer affected by the fair scheduling policy.

A series of thoughts raised by "How to run a hive SQL statement using the specified queue (Yarn) After Beeline connection HiveServer2"