Hive CliDriver hack

關於 CliDriver, 參考 Hive源碼分析：CLI入口類

這個入口天生是為 Hive 的 shell 提供的，當我在自己的應用裡想提交一個 Hive 任務時，卻發現不能直接使用（之前 MR 的 RunJar 就可以）。

正如上面的 Hive 源碼分析講的， CliDriver 做了很多的工作，那我只能 hack 一下了。

拷貝了 CliDriver 的源碼後，要做的工作有
hack log4j 為了使用自己的配置 重新定義輸出資料流以擷取執行 HQL 的結果

hack log4j 很容易做到，有這麼一段代碼，重新初始化了log4j

    boolean logInitFailed = false;    String logInitDetailMessage;    try {      logInitDetailMessage = LogUtils.initHiveLog4j();    } catch (LogInitializationException e) {      logInitFailed = true;      logInitDetailMessage = e.getMessage();    }

這個導致我們在外邊無論怎麼搗騰都沒法配置日誌系統，注釋掉這段代碼就 OK 了。

重新定義輸出資料流需要看這裡

    CliSessionState ss = new CliSessionState(new HiveConf(SessionState.class));    ss.in = System.in;    try {      ss.out = new PrintStream(System.out, true, "UTF-8");      ss.info = new PrintStream(System.err, true, "UTF-8");      ss.err = new CachingPrintStream(System.err, true, "UTF-8");    } catch (UnsupportedEncodingException e) {      return 3;    }

重新定義下

ss.out = new YourHivePrintStream();

這些做完後，並不能執行需要 MR 的Hive, 問題是你必須解決 UGI 的衝突，不然你會遇到各種沒有許可權的異常，比如

[25-11:04:58,499] [ERROR] [main] [hive.ql.Driver] Authorization failed:No privilege 'Select' found for inputs { database:db, table:tb}. Use show grant to get more details.

這裡有一個大坑，你想找到許可權異常的原因，根本無法理解這個邏輯，許可權這個東西是在 HDFS 端定義好的，登入到 HDFS 看到許可權配置都正常的啊，而且直接使用 Hive 命令列都能正常執行的好吧 

最後，盯著日誌從頭看，發現有個警告

[25-11:04:53,106] [WARN ] [main] [hadoop.security.UserGroupInformation] No groups available for user gdpi[25-11:04:53,108] [WARN ] [main] [hadoop.security.UserGroupInformation] No groups available for user gdpi

還連續警告了兩次，只能查看源碼來找原因了，首先找到這句警告資訊的出處

  public synchronized String[] getGroupNames() {    ensureInitialized();    try {      List<String> result = groups.getGroups(getShortUserName());      return result.toArray(new String[result.size()]);    } catch (IOException ie) {      LOG.warn("No groups available for user " + getShortUserName());      return new String[0];    }  }

是在 UserGroupInformation 中找到的，然後一層層找到 Hive 中

CliDriver：

    SessionState.start(ss);    // execute cli driver work    int ret = 0;    try {      ret = executeDriver(ss, conf, oproc);    } catch (Exception e) {      ss.close();      throw e;    }

看看 SessionState.start(ss) 做了什麼

    try {      startSs.authenticator = HiveUtils.getAuthenticator(          startSs.getConf(),HiveConf.ConfVars.HIVE_AUTHENTICATOR_MANAGER);      startSs.authorizer = HiveUtils.getAuthorizeProviderManager(          startSs.getConf(), HiveConf.ConfVars.HIVE_AUTHORIZATION_MANAGER,          startSs.authenticator);      startSs.createTableGrants = CreateTableAutomaticGrant.create(startSs          .getConf());    } catch (HiveException e) {      throw new RuntimeException(e);    }

HiveUtils.getAuthenticator() 擷取配置的授權管理員的類名，然後執行個體化

if (cls != null) {        ret = ReflectionUtils.newInstance(cls, conf);      }

執行個體化就執行個體化唄，但是居然又調用了

setConf(result, conf);

  public static void setConf(Object theObject, Configuration conf) {    if (conf != null) {      if (theObject instanceof Configurable) {        ((Configurable) theObject).setConf(conf);      }      setJobConf(theObject, conf);    }  }

授權管理員預設值是org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator， 它的setConf 是這樣實現的

  @Override  public void setConf(Configuration conf) {    this.conf = conf;    UserGroupInformation ugi = null;    try {      ugi = ShimLoader.getHadoopShims().getUGIForConf(conf);    } catch (Exception e) {      throw new RuntimeException(e);    }    if (ugi == null) {      throw new RuntimeException(          "Can not initialize HadoopDefaultAuthenticator.");    }    this.userName = ShimLoader.getHadoopShims().getShortUserName(ugi);    if (ugi.getGroupNames() != null) {      this.groupNames = Arrays.asList(ugi.getGroupNames());    }  }

好吧，調用了兩次

ugi.getGroupNames()

原因就是沒有擷取到期望的使用者組，因為在我的環境雷根本就不存在這個使用者（使用者身份的問題參見前面一篇文章【 Hadoop UserGroupInformation 的那些 login】）。 再看看 UGI 擷取使用者組的途徑

  public synchronized String[] getGroupNames() {    ensureInitialized();    try {      List<String> result = groups.getGroups(getShortUserName());      return result.toArray(new String[result.size()]);    } catch (IOException ie) {      LOG.warn("No groups available for user " + getShortUserName());      return new String[0];    }  }

這個是依賴於 org.apache.hadoop.security.Groups#getGroups()

  public List<String> getGroups(String user) throws IOException {    // No need to lookup for groups of static users    List<String> staticMapping = staticUserToGroupsMap.get(user);    if (staticMapping != null) {      return staticMapping;    }    // Return cached value if available    CachedGroups groups = userToGroupsMap.get(user);    long startMs = Time.monotonicNow();    // if cache has a value and it hasn't expired    if (groups != null && (groups.getTimestamp() + cacheTimeout > startMs)) {      if(LOG.isDebugEnabled()) {        LOG.debug("Returning cached groups for '" + user + "'");      }      return groups.getGroups();    }    // Create and cache user's groups    List<String> groupList = impl.getGroups(user);    long endMs = Time.monotonicNow();    long deltaMs = endMs - startMs ;    UserGroupInformation.metrics.addGetGroups(deltaMs);    if (deltaMs > warningDeltaMs) {      LOG.warn("Potential performance problem: getGroups(user=" + user +") " +          "took " + deltaMs + " milliseconds.");    }    groups = new CachedGroups(groupList, endMs);    if (groups.getGroups().isEmpty()) {      throw new IOException("No groups found for user " + user);    }    userToGroupsMap.put(user, groups);    if(LOG.isDebugEnabled()) {      LOG.debug("Returning fetched groups for '" + user + "'");    }    return groups.getGroups();  }

關鍵之處在

    // Create and cache user's groups    List<String> groupList = impl.getGroups(user);

這個 impl 是在 Groups 執行個體化時被初始化的

  public Groups(Configuration conf) {    impl =       ReflectionUtils.newInstance(          conf.getClass(CommonConfigurationKeys.HADOOP_SECURITY_GROUP_MAPPING,                         ShellBasedUnixGroupsMapping.class,                         GroupMappingServiceProvider.class),           conf);        cacheTimeout =       conf.getLong(CommonConfigurationKeys.HADOOP_SECURITY_GROUPS_CACHE_SECS,           CommonConfigurationKeys.HADOOP_SECURITY_GROUPS_CACHE_SECS_DEFAULT) * 1000;    warningDeltaMs =      conf.getLong(CommonConfigurationKeys.HADOOP_SECURITY_GROUPS_CACHE_WARN_AFTER_MS,        CommonConfigurationKeys.HADOOP_SECURITY_GROUPS_CACHE_WARN_AFTER_MS_DEFAULT);    parseStaticMapping(conf);    if(LOG.isDebugEnabled())      LOG.debug("Group mapping impl=" + impl.getClass().getName() +           "; cacheTimeout=" + cacheTimeout + "; warningDeltaMs=" +          warningDeltaMs);  }

所以，存在一個提供使用者組映射服務的工具，是有預設值的，代碼裡預設是 org.apache.hadoop.security.ShellBasedUnixGroupsMapping， 而在 core-site.xml 裡預設值是 org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback. 不管如何，這個使用者組是依賴於當前作業系統的，必須 hack. 我就實現了一個

public class MyUserGroupsMapping implements GroupMappingServiceProvider {    @Override    public List<String> getGroups(String user) throws IOException {        return Lists.newArrayList(user);    }    @Override    public void cacheGroupsRefresh() throws IOException {        // does nothing in this provider of user to groups mapping    }    @Override    public void cacheGroupsAdd(List<String> groups) throws IOException {        // does nothing in this provider of user to groups mapping    }}

然後修改了 core-site.xml . 終於成功提交的 Hive 所需執行的 MR 作業，但是為什麼都是 Failed  去查看叢集上的日誌，原來是 ClassNotFound , 我的自訂實作類別 MyUserGroupsMapping 又不在叢集上 那麼，只有偷梁換柱之策了，先修改配置指定用我的 GroupMappingServiceProvider，待本地 Hive 準備提交 MR 之前，再恢複原貌

        // set all properties specified via command line        HiveConf conf = ss.getConf();        /*hack start*/        //set hadoop.security.group.mapping to return the group of user, cause the user does not exist        String hadoopSecurityGroupMappingClass = conf.get(CommonConfigurationKeys.HADOOP_SECURITY_GROUP_MAPPING);        conf.setClass(CommonConfigurationKeys.HADOOP_SECURITY_GROUP_MAPPING, MeepoUserGroupsMapping.class, GroupMappingServiceProvider.class);        console.printInfo("set HADOOP_SECURITY_GROUP_MAPPING......");        resetGroupsMapping(conf);        /*hack end*/        for (Map.Entry<Object, Object> item : ss.cmdProperties.entrySet()) {            conf.set((String) item.getKey(), (String) item.getValue());            ss.getOverriddenConfigurations().put((String) item.getKey(), (String) item.getValue());        }        // read prompt configuration and substitute variables.        prompt = conf.getVar(HiveConf.ConfVars.CLIPROMPT);        prompt = new VariableSubstitution().substitute(conf, prompt);        prompt2 = spacesForString(prompt);        SessionState.start(ss);        /*hack start*/        //prevent submit mr to rm with my mapping class value, restore the old value        if (StringUtils.isEmpty(hadoopSecurityGroupMappingClass)) {            conf.unset(CommonConfigurationKeys.HADOOP_SECURITY_GROUP_MAPPING);        } else {            conf.set(CommonConfigurationKeys.HADOOP_SECURITY_GROUP_MAPPING, hadoopSecurityGroupMappingClass);        }        console.printInfo("reset HADOOP_SECURITY_GROUP_MAPPING......");        resetGroupsMapping(conf);        /*hack end*/        // execute cli driver work        int ret = 0;        try {            ret = executeDriver(ss, conf, oproc);        } catch (Exception e) {            ss.close();            throw e;        }        ss.close();

這裡還有個陷阱，光改變 conf 是遠遠不夠的，還需要這個

    private void resetGroupsMapping(Configuration conf) {        console.printInfo(CommonConfigurationKeys.HADOOP_SECURITY_GROUP_MAPPING + ": " +                conf.get(CommonConfigurationKeys.HADOOP_SECURITY_GROUP_MAPPING));        Groups.getUserToGroupsMappingServiceWithLoadedConfiguration(conf);        UserGroupInformation.setConfiguration(conf);    }

只有這樣，才會清空緩衝，重新 

Returning fetched groups

done.