Spark SQL UDF

Currently Spark SQL does not support custom UDFs, catalyst for the underlying SQL engine.

In SqlContext there is an analyzer to give a emptyfunctionregistry, if the SQL engine function can not be found, will be found in this functionregistry

Lookup in Emptyfunctionregistry just throws an exception.

So I've customized a functionregistry, SqlContext

@transient  protected[sql]lazyval analyzer:analyzer =    newanalyzer (catalog, Emptyfunctionregistry, CaseSensitive =true)

Class Udfrgistry extends Functionregistry {    def lookupfunction (name:string, children:seq[expression]): Expression = {      name.tolowercase match {case        ' col_set ' =>collect (Children (0)) Case        "Array" =>array (children (0) Case        "contains" =>contains (children) Case        _ = = throw new Unsupportedoperationexception      }    }  } class Sparksqlcontext (Val spctx:sparkcontext) extends SqlContext (spctx) {  @transient  override lazy Val Analyzer:analyzer =    New Analyzer (catalog, new UDF. Udfrgistry, casesensitive = True)}

This will allow you to find a custom function.

Spark SQL UDF