Hive Metastore objectstore PersistenceManager automatically closed bugs

Recently tested Hcatalog, because Hcatalog itself is a stand-alone jar package, although it can also run service, but in fact this service is Metastore thrift Server, When we write the MapReduce job based on Hcatalog, just add the Hcatalog jar package and the corresponding Hive-site.xml file to Libjars and Hadoop_classpath. However, during the test still encountered some problems, hive Metastore server after running for some time will throw the following error

2013-06-19 10:35:51,718 ERROR Server.  
Tthreadpoolserver (TThreadPoolServer.java:run)-Error occurred during processing of message. Javax.jdo.JDOFatalUserException:Persistence Manager has been closed at Org.datanucleus.jdo.JDOPersistenceManage R.assertisopen (jdopersistencemanager.java:2124) at Org.datanucleus.jdo.JDOPersistenceManager.currentTransaction ( jdopersistencemanager.java:315) at Org.apache.hadoop.hive.metastore.ObjectStore.openTransaction (Objectstore.java : 294) at Org.apache.hadoop.hive.metastore.ObjectStore.getTable (objectstore.java:732) at Sun.reflect.Na Tivemethodaccessorimpl.invoke0 (Native method) at Sun.reflect.NativeMethodAccessorImpl.invoke (Nativemethodaccesso  
        rimpl.java:39) at Sun.reflect.DelegatingMethodAccessorImpl.invoke (delegatingmethodaccessorimpl.java:25) At Java.lang.reflect.Method.invoke (method.java:597) at Org.apache.hadoop.hive.metastore.RetryingRawStore.invok E (retryingrawstore.java:111 com.sun.proxy. $Proxy 5.getTable (Unknown Source) at Org.apache.hadoop.hive.meta Store. Hivemetastore$hmshandler.get_table (hivemetastore.java:982) at Org.apache.hadoop.hive.metastore.api.ThriftHiveMet Astore$processor$get_table.getresult (thrifthivemetastore.java:5017) at Org.apache.hadoop.hive.metastore.api.Thri Fthivemetastore$processor$get_table.getresult (thrifthivemetastore.java:5005) at Org.apache.thrift.ProcessFunctio N.process (processfunction.java:32) at org.apache.thrift.TBaseProcessor.process (tbaseprocessor.java:34)

Where PersistenceManager is responsible for controlling a set of persistent objects, including creating persistent objects and querying objects, is an instance variable of ObjectStore, each objectstore has a PM, Rawstore is an interface class that interacts with the Metastore logical layer and the physical underlying metabase (such as Derby), and ObjectStore is the default implementation class for Rawstore. Hive Metastore server starts with a tprocessor, wraps a hmshandler, has a threadlocal<rawstore> instance variable inside it, Each thread maintains a rawstore

Private final threadlocal<rawstore> Threadlocalms =  
  new threadlocal<rawstore> () {  
    @Override
    Protected synchronized Rawstore InitialValue () {return  
      null;  
    }  
  };

Each request from the Hive Metastore client is assigned a workerprocess from the thread pool, and each method in Hmshandler takes Getms Rawstore () to do the exact operation.

Public Rawstore Getms () throws Metaexception {  
  Rawstore ms = Threadlocalms.get ();  
  if (ms = = NULL) {  
    ms = Newrawstore ();  
    Threadlocalms.set (MS);  
    ms = Threadlocalms.get ();  
  }  
  return MS;  
}

I can see that. Rawstore is deferred loading, which is bound to the threadlocal variable after initialization for later reuse

Private Rawstore Newrawstore () throws Metaexception {  
  log.info (addprefix ("Opening Raw store with Implemenation class : "
      + rawstoreclassname)";  
  Configuration conf = getconf ();  
      
  Return Retryingrawstore.getproxy (hiveconf, Conf, Rawstoreclassname, Threadlocalid.get ());  
}

Rawstore uses dynamic proxy mode (inheriting the Invocationhandler interface), implements the Invoke function internally, and executes the real logic through Method.invoke (), which benefits from the Method.invoke () Add your own other logic to the context, Retryingrawstore is the effect of retrying by capturing the exception thrown by the Invoke function. Due to the use of reflection mechanism, the exception is wrap in InvocationTargetException, but in hive 0.9 unexpectedly in the capture of this anomaly directly after the throw out, rather than retry, obviously wrong ah. I modified it, took out wrap target exception, judged whether instance of Jdoexception, and then handled it accordingly.

@Override public Object Invoke (object proxy, Method method, object[] args) throws Throwable {object ret = null;  
  Boolean gotnewconnecturl = false;  
  Boolean reloadconf = Hiveconf.getboolvar (hiveconf, HiveConf.ConfVars.METASTOREFORCERELOADCONF);  
      
  Boolean reloadconfonjdoexception = false;  
  if (reloadconf) {Updateconnectionurl (getconf (), NULL);  
  int retrycount = 0;  
  Exception caughtexception = null;  
      while (true) {try {if (reloadconf | | gotnewconnecturl | | reloadconfonjdoexception) {INITMS ();  
      ret = Method.invoke (base, args);  
    Break  
    catch (Javax.jdo.JDOException e) {caughtexception = (javax.jdo.JDOException) e.getcause ();  
    catch (Undeclaredthrowableexception e) {throw e.getcause ();  
      catch (InvocationTargetException e) {throwable t = e.gettargetexception (); if (t instanceof jdoexception) {caughtexception = (jdoexception) e.gettargetexception ();  
        Reloadconfonjdoexception = true;  
      Log.error ("Rawstore jdoexception:" + caughtexception.tostring ());  
      }else {throw e.getcause ();  
    } if (RetryCount >= retrylimit) {throw caughtexception;  
    ASSERT (RetryInterval >= 0);  
    retrycount++; Log.error (String.Format ("JDO datastore error.) Retrying metastore command "+" after%d ms (attempt%d of%d), RetryInterval, RetryCount, Retrylimit)  
    ;  
    Thread.Sleep (RetryInterval); If we have a connection error, the JDO connection URL hook might//provide us with a new URL to access the Datas  
    Tore.  
    String Lasturl = getConnectionUrl (getconf ());  
  Gotnewconnecturl = Updateconnectionurl (getconf (), lasturl);  
return ret; }