Variable delivery settings for hive

Today colleagues in Oozie workflow to execute a hive query, but directly to the exception: Variable substitution depth too large:40, from the online query can be confirmed that the statement is caused by the use of too many variables, In previous versions of Hive, this limit was written to 40 dead, querying hive for the latest original code, although the location of the tips information has changed, but the same principle:

### org.apache.hadoop.hive.ql.parse.VariableSubstitution:

  public string substitute (hiveconf conf, string  Expr)  {    if  (expr == null)  {       return expr;    }    if  (Hiveconf.getboolvar (Conf,  confvars.hivevariablesubstitute))  {      l4j.debug ("Substitution  is on:  " + expr);    } else {       return expr;    }    int depth =  Hiveconf.getintvar (conf, confvars.hivevariablesubstitutedepth);    return  Substitute (conf, expr, depth);   }123456789101112 

If Hive.variable.substitute is turned on (default on), use Systemvariables's Substitute method and hive.variable.substitute.depth (default = 40) For further judgment:

  protected final string substitute (configuration conf, string expr,  int depth)  {    matcher match = varpat.matcher ("");     string eval = expr;    stringbuilder builder =  new stringbuilder ();     int s = 0;    for   (;  s <= depth; s++)  {      match.reset (eval);       builder.setlength (0);       int prev  = 0;      boolean found = false;       while  (Match.find (prev))  {        string  group = match.group ();         string var =  group.substring (2, gRoup.length ()  - 1);  // remove ${   }        string substitute = getsubstitute (Conf,  var);        if  (substitute == null)  {           substitute = group;   //  append as-is        } else {           found = true;        }         builder.append (Eval.substring (Prev, match.start ())). Append (substitute);         prev = match.end ();       }      if  (!found)  {         return eval;      }      builder.append (eval.substring (prev));       eval = builder.tostring ();    }    if  (s  > depth)  {      throw new illegalstateexception (            "variable substitution depth is  deeper than  " + depth + "  for expression  " + expr";     }    return eval;  }  12345678910111213141516171819202122232425262728293031323334

If the ${} argument is used that exceeds the number of hive.variable.substitute.depth, the exception is thrown directly, so we precede the statement with the set hive.variable.substitute.depth=100; Problem solved!

The execution of the set command is performed specifically in the Commandprocessor implementation class Setprocessor, but the substitute statement is also invoked in Compileprocessor, which is called when the hive statement is compiled. Therefore, when Oozie calls Beeline to execute a statement when it is used, the compile stage reports an exception.

But why does hue have no problem executing this statement directly? Because Hue uses python-developed beeswax in the execution of Hive, and beeswax directly processes these variables, replaces them with the actual values of the variables and submits them to hive for execution:

def substitute_variables (Input_data, substitutions): "" "replaces variables with the values from substitutions. "" "Def f (value): If not isinstance (value, basestring): return value New_value = Template (value). Safe_substitu    TE (substitutions) if new_value! = Value:LOG.debug ("Substituted%s and%s"% (repr (value), repr (New_value))) Return New_value return Recursive_walk (F, Input_data)

Variable delivery settings for hive