Eighth article: Spark SQL Catalyst Source Analysis UDF

/** Spark SQL Source Analysis series Article */

In the world of SQL, in addition to the commonly used processing functions provided by the official, extensible external custom function interface is generally provided, which has become a fact of the standard.

In the previous article on the core process of spark SQL source analysis, the role of the Spark SQL Catalyst Analyzer has been introduced, which includes the function of the resolvefunctions function. However, with the release of the Spark1.1 release, the Spark SQL code has a lot of new improvements and new features, and my previous 1.0-based source analysis is somewhat different, such as support UDF:

spark1.0 and previous implementations:

[Java]View PlainCopy

1. Protected[sql] Lazy val catalog:catalog = new Simplecatalog
2. @transient
3. Protected[sql] Lazy val analyzer:analyzer =
4. New Analyzer (catalog, emptyfunctionregistry, casesensitive = true) //emptyfunctionregistry NULL implementation
5. @transient
6. Protected[sql] Val optimizer = Optimizer

Spark1.1 and future implementations:

[Java]View PlainCopy

1. Protected[sql] Lazy val functionregistry:functionregistry = new Simplefunctionregistry // Simplefunctionregistry implementation, support for simple UDFs
3. @transient
4. Protected[sql] Lazy val analyzer:analyzer =
5. New Analyzer (catalog, functionregistry, casesensitive = true)

First, Primer:

For functions in SQL statements, they are parsed into unresolvedfunction by Sqlparser. Unresolvedfunction will eventually be analyzed by analyzer.

Sqlparser:

In addition to unofficially defined functions, you can also define custom functions that SQL parser will parse.

[Java]View PlainCopy

1. Ident ~ "(" ~ repsep (expression, ",") <~ ")" ^^ {
2. Case Udfname ~ _ ~ Exprs = Unresolvedfunction (Udfname, Exprs)

Encapsulates the Sqlparser incoming udfname and Exprs into a class class unresolvedfunction inherited from expression.

Just this expression of datatype and so on a series of properties and eval calculation method can not access, forced access will throw an exception, because it is not resolved, just a vector.

[Java]View PlainCopy

2. Case class Unresolvedfunction (name:string, children:seq[expression]) extends Expression {
4. Override Def DataType = throw new Unresolvedexception (This, "DataType")
6. Override def foldable = throw new Unresolvedexception (This, "foldable")
8. Override Def nullable = throw new Unresolvedexception (This, "nullable")
10. Override lazy Val resolved = false
14. //unresolved functions is transient at compile time and don ' t get evaluated during execution.
16. Override Def eval (input:row = null): Evaluatedtype =
18. throw New Treenodeexception (This,the S "No function to evaluate expression. Type: ${this.nodename} ")
22. Override Def toString = s"' $name (${children.mkstring (",")})"
24. }<strong></strong>

Analyzer:

When analyzer initializes, it will require catalog,database and table metadata relationships, as well as functionregistry to maintain the UDF name and metadata for the UDF implementation, using Simplefunctionregistry.

[Java]View PlainCopy

2. /**
4. * Replaces [[Unresolvedfunction]]s with concrete [[catalyst.expressions.Expression expressions]].
6. */
8. Object Resolvefunctions extends Rule[logicalplan] {
10. def apply (Plan:logicalplan): Logicalplan = Plan Transform {
12. Case Q:logicalplan =
14. Q transformexpressions { //transformexpressions operation for current Logicalplan
16. Case u @ unresolvedfunction (name, children) if u.childrenresolved =/ /if traverse to Unresolvedfunction
18. Registry.lookupfunction (name, children) //Find UDF functions from the UDF metadata table
20. }
22. }
24. }

Second, UDF registration

2.1 Udfregistration

registerfunction ("Len", (x:string) =>x.length)

registerfunction is udfregistration under the method, SqlContext now implement udfregistration this trait, as long as import sqlcontext, that is, you can use the UDF function.

udfregistration Core method Registerfunction:

Registerfunction method Signature Def Registerfunction[t:typetag] (name:string, Func:function1[_, T]): Unit

Accept a udfname and a functionn, which can be Function1 to Function22. That is, the UDF parameter only supports 1-22. (Scala's pain.)

The internal builder constructs an expression through SCALAUDF, where scalaudf inherits from expression (it is simple to understand that the current simpleudf is a catalyst's expression), The function that passes in to Scala is implemented as a UDF, and the reflection checks whether the field type is allowed by catalyst, see scalareflection.

[Java]View PlainCopy

1. def Registerfunction[t:typetag] (name:string, Func:function1[_, T]): Unit = {
2. def builder (e:seq[expression]) = Scalaudf (func, Scalareflection.schemafor (Typetag[t]). DataType, E)// Construct Expression
3. Functionregistry.registerfunction (name, builder)//SqlContext functionregistry (Maintenance of a hashmap to manage UDF mappings) Registration

2.2 Register Function:

Note: Here Functionbuilder is a type Functionbuilder = seq[expression] = Expression

[Java]View PlainCopy

2. Class Simplefunctionregistry extends Functionregistry {
4. Val functionbuilders = new mutable. Hashmap[string, Functionbuilder] () //udf mapping Relationship maintenance [udfname,expression]
8. def registerfunction (name:string, Builder:functionbuilder) = { //put expression into map
10. Functionbuilders.put (name, builder)
12. }
16. Override Def lookupfunction (Name:string, children:seq[expression]): Expression = {
18. Functionbuilders (name) (children) //Find UDF, return expression
20. }
22. }

At this point, we register a Scala function as one of the catalyst's expression, which is the simple UDF of Spark.

Third, UDF calculation:

Since the UDF has been encapsulated as an expression node in the catalyst tree, it is calculated as the Eval method of the scalaudf.

The parameters required by the function are computed by row and expression, and the function is called by reflection to achieve the purpose of calculating the UDF.

Scalaudf inherits from Expression:

SCALAUDF accepts a function, DataType, and a series of expressions.

Simple, look at the comments can:

[Java]View PlainCopy

2. Case class scalaudf (Function:anyref, Datatype:datatype, children:seq[expression])
4. extends Expression {
8. Type Evaluatedtype = any
12. def nullable = true
16. Override Def toString = s"scalaudf (${children.mkstring (",")})"
18. Override Def eval (input:row): any = {
20. Val result = children.size Match {
22. Case 0 = function.asinstanceof[() = any] ()
24. case 1 = function.asinstanceof[(any) + any] (children (0). Eval (input)) //Reflection Call function
26. Case 2 =
28. function.asinstanceof[(Any, any) and any] (
30. Children (0). Eval (input), //Expression argument calculation
32. Children (1). Eval (input)
34. case 3 =
36. function.asinstanceof[(Any, any, any) = + any] (
38. Children (0). Eval (input),
40. Children (1). Eval (input),
42. Children (2). Eval (input)
44. case 4 =
46. ......
48. Case x = //scala function only supports 22 parameters, enumerated here.
50. function.asinstanceof[(Any, any, any, any, any, any, any, any, any, any, any, any, any, any, any, any, any, any, Any, and any) = any] (
52. Children (0). Eval (input),
54. Children (1). Eval (input),
56. Children (2). Eval (input),
58. Children (3). Eval (input),
60. Children (4). Eval (input),
62. Children (5). Eval (input),
64. Children (6). Eval (input),
66. Children (7). Eval (input),
68. Children (8). Eval (input),
70. Children (9). Eval (input),
72. Children (). Eval (input),
74. Children (one). Eval (input),
76. Children (). Eval (input),
78. Children (). Eval (input),
80. Children. Eval (input),
82. Children (). Eval (input),
84. Children (+). Eval (input),
86. Children (+). Eval (input),
88. Children. Eval (input),
90. Children. Eval (input),
92. Children (). Eval (input),
94. Children (+). Eval (input)

Iv. Summary

The current UDF for Spark is actually Scala function. Encapsulates the Scala function into a catalyst expression, which computes the current input row using the same eval method for SQL calculations.

Writing a spark UDF is very simple, just give the UDF a function name and pass a Scala function. The ability to rely on Scala's functional programming makes it easier to write Scala UDFs, and more understandable than a hive UDF.

--eof--

Original articles, reproduced please specify:

Reprinted from: Oopsoutofmemory Shengli's blog, oopsoutofmemory

This article link address: http://blog.csdn.net/oopsoom/article/details/39395641

Note: This document is based on the attribution-NonCommercial use-prohibition of the deduction of the 2.5 China (CC by-nc-nd 2.5 CN) Agreement, which is welcome to reprint, forward and comment, but please retain the author's attribution and link to the article. Please contact me if you need to negotiate for commercial purposes or in connection with licensing.

Transferred from: http://blog.csdn.net/oopsoom/article/details/39395641

Eighth article: Spark SQL Catalyst Source Analysis UDF