Problems encountered in implementing kerberos Authentication and impersonation on hive server of hive0.11

Background recently working on upgrading hive0.9 to 0.11, one of the steps is to apply the patch re-apply previously applied to 0.9 to 0.11, there is a patch (https://github.com/lalaguozhe/hive/commit/f2892f9e4706f3ea04117cbc7e7f54ff6af1e415) according to the implementation of hive metastore service, sasl kerberos authentication is added to hive server, you can use impersonate as the client ugi identity to start a job. (The hive service daemon user identity is used by default. As a result, all queries share one user to start a job ). But after re-apply patch, we found that using jdbc client to access hive server is a null result set returned by some statements (fetchedRows in HiveQueryResultSet is an empty set ), no error was reported. It is very strange that the statement with only one case will return the result normally after multiple attempts, that is, the statement similar to "select * from xxx where yyy" will not start MapReduce Job, other statements such as "show tables/databases" and "select a from xxx" return empty result sets. The following process is performed when the Hive jdbc client (thrift client at the underlying layer) submits a statement: 1. construct the execute_args object, which encapsulates the statements to be executed, send the remote method name execute and execute_args object, and receive the result returned by execute. At this time, the client has obtained the column names and column types information. [Java] public void send_execute (String query) throws org. apache. thrift. TException {execute_args args = new execute_args (); args. setQuery (query); sendBase ("execute", args);} the server is processed by HiveServerHandler. The corresponding execute method will generate a new driver and call the driver. run (cmd) to execute the specific statement 2. send the remote method fetchN multiple times and the maximum number of returned records numRows. The returned result assembly is placed in List <String> fetchedRows. For example, if a total of 90 records are returned and each fetchN returns a maximum of 50 records, two fetchN [java] public void send_fetchN (int numRows) throws org are called. apache. thrift. TException {fetchN_args args = new fetchN_args (); args. setNumRows (numRows); sendBase ("fetchN", args);} The fetchN in the HiveServerHandler on the server calls the driver. getResult (): The FetchOperator in the FetchTask of QueryPlan gets the file path storing the result set and obtains the InputFormat information. With InputFormat, you can call the getSplits method to obtain an InputSplit string and obtain each InputSplit in sequence. Iterator next to get the key/value (value indicates the record of each row) 3. hiveConnection close, send the clean remote method name [java] public void send_clean () throws org. apache. thrift. TException {clean_args args = new clean_args (); sendBase ("clean", args);} the server executes the clean method, closes the driver, and cleans the context, delete the scratch directories generated by the statement (both local file system and hdfs are cleared) Context. java removeScratchDir method [java] private void removeScratchDir () {(Map. entry <String, String> entry: fsScratchDirs. entrySet () {try {Path p = new Path (entry. getValue (); p. getFileSystem (conf ). delete (p, true);} catch (Exception e) {LOG. warn ("Error Removing Scratch:" + StringUtils. stringifyException (e) ;}} fsScratchDirs. clear ();} specific positioning statements can be divided into three types: 1. for MR Job statements such as "select * from xxx", the server directly uses MetastoreClient to obtain the hdfs storage path corresponding to the table and read it using FetchTask. Note that a configuration item "hive. fetch. task. conversion ", introduced by jira HIVE-887. The default value is minimal. In addition, it can be set to more. In minimal mode, for" select star, FILTER on partition columns, LIMIT only "does not start MR Job. In more mode, for" SELECT, FILTER, LIMIT only (TABLESAMPLE, virtual columns) ", there is no subquery, the aggregate operation and distinct statement do not start MR Job, greatly reducing query latency. Observe the implementation code. In fact, it uses TableScanOperator, FilterOperator, as the Sub-Operator of FetchOperator, SelectOperator obtains the data to the client (that is, hive The Operator Tree information is saved in the filter or projectionFetchOperator, similar to calling the operator. process () method in depth traversal. FetchOperator. java [java] public boolean pushRow () throws IOException, HiveException {InspectableObject row = getNextRow (); if (row! = Null) {operator. process (row. o, 0);} return row! = Null;} For example, for the statement "select c1 from abc where c1 = 1;", Fetch Operator-> TableScanOperator-> FilterOperator-> SelectOperator-> ListSinkOperator 2 will be called in sequence. a ddl/dmlsentence similar to "show tables/databases" will first create a scratch (by hive.exe c. local. scratchdir configuration), write the calculation result to the local scratch directory, and then read 3 through FetchTask. similar to "select count (1) from tblname", the MR job's syntax starts, and the scratch (by hive.exe c. scratchdir configuration). The calculation result is written to the hdfs scratch object. And then read it through FetchTask. After multiple attempts, the first type of statements can return results, and the second and third types of statements return NULL sets, the difference between the two lies in whether to directly read the original table data path or read from the scratch directory. HIVE sets the environment Context in the Compile stage to create the local/hdfs scratch directory. Before Version 0.10, there will be a problem. If the user forces to kill the statements being executed, then these scratch dir will become orphaned dir, which has not been cleared. HIVE added HIVE-0.10 in 3251 to solve this problem. Set HDFSCleanUp of Context in Driver to true [java] command = new VariableSubstitution (). substitute (conf, command); ctx = new Context (conf); ctx. setTryCount (getTryCount (); ctx. setCmd (command); ctx. setHDFSCleanup (true); Add the Scratch dir path to the deleteOnExit set in filesystem instance [java] private String getScratchDir (String scheme, String authority, boolean mkdir, string scratchDir) {String fileSys Tem = scheme + ":" + authority; String dir = fsScratchDirs. get (fileSystem); if (dir = null) {Path dirPath = new Path (scheme, authority, scratchDir); if (mkdir) {try {FileSystem fs = dirPath. getFileSystem (conf); dirPath = new Path (fs. makeQualified (dirPath ). toString (); if (! Fs. mkdirs (dirPath) {throw new RuntimeException ("Cannot make directory:" + dirPath. toString ();} if (isHDFSCleanup) {fs. deleteOnExit (dirPath) ;}} catch (IOException e) {throw new RuntimeException (e) ;}} dir = dirPath. toString (); fsScratchDirs. put (fileSystem, dir);} When filesystem is closed, all files marked deleteOnExit [java] public void close () throws IOException {// delete all files that were Marked as delete-on-exit. processDeleteOnExit (); CACHE. remove (this. key, this);} at the same time, we know that the FileSystem abstract class has a static final member variable Cache. The key is cached by the combination of schema, authority, ugi, and unique, there is also a ClientFinalizer object (which implements Runnable) registered to the shutdown hook of the JVM. When the JVM is disabled, the ClientFinalizer thread will be started to close all filesystems in the Cache in sequence, this method is used to clear and delete the resource files that are linked to filesystem. In Hive, these files are local/hdfs scratch private class ClientFinalizer implements. Runnable {@ Override public synchronized void run () {try {closeAll (true);} catch (IOException e) {LOG.info ("FileSystem. cache. closeAll () threw an exception: \ n "+ e) ;}} returned to the previous question, the second and third types of statements can be used to log in the execute method of the hive server to confirm that the data has been dumped to the scratch directory, but when the fetchN method is entered, these files are inexplicably lost, resulting in empty data being read. After checking for a long time, we found that jira hive-0.10 was introduced in HIVE 3098 to solve the problem of FileSystem Cache memory leakage. Each function call will be handled by a TUGIAssumingProcessor in HadoopThriftAuthBridge20S. In the process method, a proxyUser UGI will be created first and the clientUgi will be used. doAs to execute the specific logic, so that daemon user will impersonate the client user. If you want to create a filesystem object in the specific logic code, it will use UserGroupInformation. getCurrentUser () (clientUgi) is added to the Cache as part of the FileSystem Cache Key. HIVE-3098 adds the finally of the process Method to clear the FileSystem instance [java] finally {if (clientUGI! = Null) {// clear the filesystem try {FileSystem. closeAllForUGI (clientUgi);} catch (IOException exception) {LOG. error ("cocould not clean up file-system handles for UGI:" + clientUgi, exception) ;}} it is because the first execute method calls FileSystem in finally. closeAllForUGI (clientUgi), close the related filesystem object, and delete the bound scratch directory. The second fetchN method does not read data. But why does hive server 2 that implements kerberos Authentication and impersonation not encounter this problem? In fact, the impersonation (set hive. server2.enable. doAs = true) is not at the thrift processor level but at the hive session level for impersonation, so that useProxy = false in filesystem [java] // hive server 2 is not cleared in process finally; if (useProxy) {clientUgi = UserGroupInformation. createProxyUser (endUser, UserGroupInformation. getLoginUser (); remoteUser. set (clientUgi. getaskusername (); returnCode = clientUgi. doAs (new Priv IlegedExceptionAction <Boolean> () {public Boolean run () {try {return wrapped. process (inProt, outProt);} catch (TException te) {throw new RuntimeException (te) ;}});} else {remoteUser. set (endUser); return wrapped. process (inProt, outProt);} In HiveSessionProxy (proxy HiveSessionImplwithUGI), use ugi doAs to execute [java] public Object invoke (Object arg0, final Method method, final Object [] args) throws Throwab Le {try {return ShimLoader. getHadoopShims (). doAs (ugi, new PrivilegedExceptionAction <Object> () {@ Override public Object run () throws HiveSQLException {try {return method. invoke (base, args);} catch (InvocationTargetException e) {if (e. getCause () instanceof HiveSQLException) {throw (HiveSQLException) e. getCause ();} else {throw new RuntimeException (e. getCause () ;}} catch (IllegalArgumentEx Ception e) {throw new RuntimeException (e);} catch (IllegalAccessException e) {throw new RuntimeException (e) ;}});} catch (UndeclaredThrowableException e) {Throwable innerException = e. getCause (); if (innerException instanceof PrivilegedActionException) {throw innerException. getCause ();} else {throw e. getCause () ;}} the client calls HiveConnection. after close, the server will eventually call HiveSessionImplwithUGI. clo Se (); close the filesystem object [java] public void close () throws HiveSQLException {try {acquire (); ShimLoader. getHadoopShims (). closeAllForUGI (sessionUgi); cancelDelegationToken ();} finally {release (); super. close () ;}} after understanding the implementation principles, there are three solutions: 1. disable FileSystem Cache [plain] $ HIVE_HOME/bin/Hive -- service hiveserver2 -- hiveconf fs when starting hive Server. hdfs. impl. disable. cache = true -- hiveconf fs. file. impl. disabl E. cache = true 2. setHDFSCleanup (false) is set in Hive Context, so the scratch directory will not be automatically cleared, but there will be orphaned files problems, you need to deploy another scheduled script to actively delete 3. in the thrift processor, determine whether to close filesystem Based on the return values of each function call. When the last connection is closed, we actively close filesystem. Finally, we adopt the third solution: Modify the hive settings as follows [Java] finally {if (! ReturnCode) {if (clientUgi! = Null) {LOG.info ("Start to close filesystem for clientUgi:" + clientUgi. getUserName (); try {FileSystem. closeAllForUGI (clientUgi);} catch (IOException exception) {LOG. error ("cocould not clean up file-system handles for UGI:" + clientUgi, exception );}}}} at the same time, add the logic [java] public void clean () {if (driver! = Null) {driver. close (); driver. destroy ();} SessionState session = SessionState. get (); if (session. getTmpOutputFile ()! = Null) {session. getTmpOutputFile (). delete ();} pipeIn = null; try {LOG.info ("Start to close filesystem for ugi:" + UserGroupInformation. getCurrentUser (). getUserName (); ShimLoader. getHadoopShims (). closeAllForUGI (UserGroupInformation. getCurrentUser ();} catch (IOException ioe) {ioe. printStackTrace () ;}} modified the above Code and re-compiled the code. The results of the previous three case statements can be returned normally. This problem has been around for a day and there are not many hive bugs, so from time to time, we will step on the pitfall, but in the process of discovering the problem to debug and then solving the problem I also learned a lot.