Research, installation and deployment of Hive Server 2

Background

We have been using Hive server 1 for a long time, and users Ad-hoc Query,hive-web, wormhole, operations tools, and so on, are submitting statements through Hive Server. But hive server is extremely unstable, often inexplicable mysterious death, causing the client side of all connection are blocked. To this we have to configure a crontab check script, will continue to execute "show tables" statement to detect whether the server suspended animation, if suspended animation, can only kill daemon process restart. In addition hive Server 1 concurrency support is not good, if a user set some environment variables in the connection, bound to a thrift worker thread, the user disconnected, another user also created a connection, He may also be assigned to the previous worker thread and will reuse the previous configuration. This is because thrift does not support detecting whether a client is disconnected, and it cannot purge session state information. It is difficult to do ha in the same way that session is bound to worker thread. Hive server 2 has the perfect support for the session, the client side each time RPC call will bring a SessionID, the Server side will be mapping to save the state of the message of session states, so that any worker Thread can execute different statements of the same session without tying them to the same one.

Hive 0.11 contains Hive server 1 and Hive server 2, and also includes 1 for backward compatibility. In the long run, Hive Server 2 will be the first choice.

Configuration

1. Configure the Hive server listening port and host

<property>  
  <name>hive.server2.thrift.port</name>  
  <value>10000</value>  
</property>  
<property>  
  <name>hive.server2.thrift.bind.host</name>  
  <value>test84.hadoop</value>  
</property>

2. Configure Kerberos authentication so that thrift client and Hive server 2, Hive server 2 and HDFs interaction are authenticated by Kerberos

<property>  
  <name>hive.server2.authentication</name>  
  <value>kerberos</value >  
  <description>  
    Client authentication types.  
       None:no authentication Check  
       ldap:ldap/ad based authentication  
       KERBEROS:KERBEROS/GSSAPI authentication Custom:custom authentication Provider (use with property  
               Hive.server2.custom.authentication.class)  
  </ description>  
</property>  
<property>  
  <name> Hive.server2.authentication.kerberos.principal</name>  
  <value>hadoop/_host@dianping.com</ value>  
</property>  
<property>  
  <name> Hive.server2.authentication.kerberos.keytab</name>  
  <value>/etc/hadoop.keytab</value>  
</property>

3. Set impersonation so that Hive server executes the statement as the submitting user, and if set to false, the statement is executed with Admin user from Hive server daemon

<property>  
  <name>hive.server2.enable.doAs</name>  
  <value>true</value>  
</property>

Execute command $hive_home/bin/hive--service hiveserver2 or $hive_home/bin/hiveserver2 Will invoke the Org.apache.hive.service.server.HiveServer2 Main method to start the

The output log information in hive log is as follows:

2013-09-17 14:59:21,081 INFO Server. HiveServer2 (HiveStringUtils.java:startupShutdownMessage (604))-startup_msg:/*********************************** startup_msg:starting HiveServer2 startup_msg:host = test84.hadoop/10.1.77.84 startup_m  
Sg:args = [] Startup_msg:version = 0.11.0 Startup_msg:classpath = slightly ......... 2013-09-17 14:59:21,957 INFO Security. Usergroupinformation (UserGroupInformation.java:loginUserFromKeytab (633))-Login successful for user hadoop/ Test84.hadoop@DIANPING.COM using keytab file/etc/hadoop.keytab 2013-09-17 14:59:21,958 INFO service.  
Abstractservice (AbstractService.java:init)-Service:operationmanager is inited. 2013-09-17 14:59:21,958 INFO Service.  
Abstractservice (AbstractService.java:init)-Service:sessionmanager is inited. 2013-09-17 14:59:21,958 INFO Service.  
Abstractservice (AbstractService.java:init)-Service:cliservice is inited. 2013-09-17 14:59:21,959 INFO Service. ABstractservice (AbstractService.java:init)-Service:thriftcliservice is inited. 2013-09-17 14:59:21,959 INFO Service.  
Abstractservice (AbstractService.java:init)-Service:hiveserver2 is inited. 2013-09-17 14:59:21,959 INFO Service.  
Abstractservice (AbstractService.java:start)-Service:operationmanager is started. 2013-09-17 14:59:21,960 INFO Service.  
Abstractservice (AbstractService.java:start)-Service:sessionmanager is started. 2013-09-17 14:59:21,960 INFO Service.  
Abstractservice (AbstractService.java:start)-Service:cliservice is started. 2013-09-17 14:59:22,007 INFO Metastore. Hivemetastore (HiveMetaStore.java:newRawStore (409))-0:opening Raw store with implemenation Class:o Rg.apache.hadoop.hive.metastore.ObjectStore 2013-09-17 14:59:22,032 INFO metastore. ObjectStore (ObjectStore.java:initialize (222))-ObjectStore, initialize called, 2013-09-17 14:59:22,955 INFO metastore. ObjectStore (ObjectStore.java:getPMF (267))-Setting metastore oBject pin classes with hive.metastore.cache.pinobjtypes= "Table,storagedescriptor,serdeinfo,partition,database,type , Fieldschema,order "2013-09-17 14:59:23,000 INFO metastore. ObjectStore (ObjectStore.java:setConf (205))-initialized ObjectStore 2013-09-17 14:59:23,909 INFO metastore. Hivemetastore (HiveMetaStore.java:logInfo (452))-0:get_databases:default 2013-09-17 14:59:23,912 INFO hivemetastore. Audit (HiveMetaStore.java:logAuditEvent (238))-ugi=hadoop/test84.hadoop@dianping.com ip=unknown-ip-addr cmd=get_ Databases:default 2013-09-17 14:59:23,933 INFO service.  
Abstractservice (AbstractService.java:start)-Service:thriftcliservice is started. 2013-09-17 14:59:23,948 INFO Service.  
Abstractservice (AbstractService.java:start)-Service:hiveserver2 is started. 2013-09-17 14:59:24,025 INFO Security. Usergroupinformation (UserGroupInformation.java:loginUserFromKeytab (633))-Login successful for user hadoop/ Test84.hadoop@DIANPING.COM using keytab File/etc/hadoOp.keytab 2013-09-17 14:59:24,047 INFO Thrift. Thriftcliservice (ThriftCLIService.java:run (435))-Thriftcliservice listening on test84.hadoop/10.1.77.84:10000