Variable delivery settings for hive

Variable delivery settings for hive

Today colleagues in Oozie workflow to execute a hive query, but directly to the exception: Variable substitution depth too large:40, from the online query can be confirmed that the statement is caused by the use of too many variables, In previous versions of Hive, this limit was written to 40 dead, querying hive for the latest original code, although the location of the tips information has changed, but the same principle:

# # # Org.apache.hadoop.hive.ql.parse.VariableSubstitution:

  publicsubstitute(HiveConf conf, String expr) {    ifnull) {      return expr;    }    if (HiveConf.getBoolVar(conf, ConfVars.HIVEVARIABLESUBSTITUTE)) {      l4j.debug("Substitution is on: " + expr);    else {      return expr;    }    int depth = HiveConf.getIntVar(conf, ConfVars.HIVEVARIABLESUBSTITUTEDEPTH);    return substitute(conf, expr, depth);  }

If Hive.variable.substitute is turned on (default on), use Systemvariables's Substitute method and hive.variable.substitute.depth (default = 40) To make further judgments:

  protected FinalStringSubstitute(Configuration conf, String expr,intDepth) {Matcher match = Varpat.matcher ("");    String eval = expr; StringBuilder Builder =NewStringBuilder ();ints =0; for(; s <= depth; s++)      {Match.reset (eval); Builder.setlength (0);intPrev =0;BooleanFound =false; while(Match.find (prev))        {String group = Match.group (); String var = group.substring (2, Group.length ()-1);//Remove ${...}String substitute = Getsubstitute (conf, Var);if(Substitute = =NULL) {substitute = Group;//Append As-is}Else{found =true;        } builder.append (Eval.substring (prev, Match.start ())). append (substitute);      prev = Match.end (); }if(!found) {returnEval      } builder.append (Eval.substring (prev));    eval = builder.tostring (); }if(s > Depth) {Throw NewIllegalStateException ("Variable substitution depth is deeper than"+ Depth +"for expression"+ expr); }returnEval }

If the ${} parameter is used that exceeds the number of hive.variable.substitute.depth, the exception is thrown directly, so we add the set hive.variable.substitute.depth=100 directly in front of the statement; Problem Solving!

The execution of the set command is performed specifically in the Commandprocessor implementation class Setprocessor, but the substitute statement is also invoked in Compileprocessor, which is called when the hive statement is compiled. Therefore, when Oozie calls Beeline to execute a statement when it is used, the compile stage reports an exception.

But why does hue have no problem executing this statement directly? Because Hue uses python-developed beeswax in the execution of Hive, and beeswax directly processes these variables, replaces them with the actual values of the variables and submits them to hive for execution:

def  substitute_variables   (Input_data, substitutions) :   def  f  Span class= "Hljs-params" > (value) :  if   Not  isinstance (value, basestring): return  value New_value = Template (valu e). Safe_substitute (substitutions) if  new_value! = Value:LOG.debug (" substituted%s,%s "% (repr (value), repr (New_value))) return  new_value return  recursive_walk (F, Input_data) 

Variable delivery settings for hive