Catalyst Optimizer Optimizer

The optimizer catalyst for Spark SQL is easy to scale. It supports both rule-based (rule-based) and cost-based (cost-based) optimization methods.

Within it, the catalyst contains a common library of rules that represent the tree and the action tree. In this framework, a library is currently implemented for relational query processing (e.g., expressions, logical query plans), and some rules for processing query execution at different stages (analysis, logical optimization, physical optimization, code generation).

Tree

The main data type of the catalyst is a tree of node objects. Each node has a node type and 0 children at most. The new node type is a subclass within Scala TreeNode .

Add(Attribute(x), Add(Literal(1), Literal(2)))

Rules

You can use rules to manipulate trees, that is, the way to convert one tree to another tree. Although rules can execute arbitrary code on the input tree (because the tree is also just a Scala object), it is common practice to use a series of pattern matching methods to find and replace subtrees with specific results.

For example, we implement the add operation between the folding (folds) constants below:

    tree.transform {      case Add(Literal(c1), Literal(c2)) => Literal(c1+c2)    }

In this way, the right x+(1+2) tree uses the rule, which creates a new tree x+3 .

In a conversion call, the rule can perform multiple match pattern.

Finally, the conditions and contents of the rule can contain arbitrary code. This allows catalyst to be simpler for novice users.

In addition, the transformation of the tree (transformation) is performed on the immutable (immutable) tree, which is easy to debug and facilitates the parallelism of the optimizer.

Using Catalyst in Spark SQL

Analysis

Whether it is an AST (abstract syntax tree) obtained by SQL parser or a Dataframe object built using the API, the relationship (relation) must contain an unresolved (unresolved) attribute reference or relationship: for example SELECT col FROM sales, we didn't know the type of col, even if Col was a valid column name, until we queried the sales table.

An attribute is called unresolved if we does not know its type or has not matched it to a input table (or an alias)

Spark SQL parses these properties using the catalyst rules and the Catalog object tables in the trace data source.

This part of the code is located org.apache.spark.sql.catalyst.analysis.Analyzer.scala .

Logical Optimizations

Logical optimizations are completely rule-based (rule-based).

Includes: constant folding (constant folding), predicate push (predicate pushdown), drop Shadow (project pruning), null propagation, bool expression simplification, and so on.

Predicata pushdown: Moves the predicate in the WHERE clause of the outer query block into the included lower-level query block (such as a view) to enable early data filtering and possibly better utilization of the index.

For different situations, adding rules is also simple.

For example, the following simplifies the like expression:

Object Likesimplification extends Rule[logicalplan] {val StartsWith ="([^_%]+)%". r val EndsWith ="%([^_%]+)". r val contains ="%([^_%]+)%". r val Equalto ="([^_%]*)". R def apply (Plan: Logicalplan): Logicalplan = plan Transformallexpressions { CaseLike (L, Literal (StartsWith (pattern), StringType)) if ! pattern. EndsWith ("\ \") =StartsWith (L, Literal (pattern)) CaseLike (L, Literal (EndsWith (pattern), stringtype)) =EndsWith (L, Literal (pattern)) CaseLike (L, Literal (contains (pattern), StringType)) if ! pattern. EndsWith ("\ \") =Contains (L, Literal (pattern)) CaseLike (L, Literal (Equalto (pattern), stringtype)) =Equalto (L, Literal (pattern))}}

The comment for this code is:
>
Simplifies like expressions that does not need full regular expressions to evaluate the condition.
For example, when the expression was just checking to see if a string starts with a given pattern.

This part of the code is located org.apache.spark.sql.catalyst.optimizer.Optimizer.scala .

Physical planning

During the physical planning phase, Spark SQL generates one or more physical plans through a logical plan, and then uses the cost model to select a plan.

Currently, the code model is only used when selecting the join algorithm: If a relationship is small, Sparksql will use the broadcast join to take advantage of its peer-to-peer broadcast characteristics. The cost model can be extended to other algorithms in the future.

The physical plan also performs rule-based optimizations.

This part of the code is located org.apache.spark.sql.execution.SparkStrategies.scala .

Code Generation

Generating Java sub-code at run time is the last stage of query optimization.

Because Spark SQL basically operates on a memory dataset, this is Cpu-bound, so the generated code can run faster.

The code generation engine is very difficult to implement, basically equivalent to a compiler. However, the new features that rely on the Scala language Quasiquotes make it much simpler. Quasiquotes allows the program to construct an abstract syntax tree (an abstract syntax trees) that is submitted to the compiler to generate the child section code at run time.

For example (x+y)+1 , if there is no code generation, such an expression would traverse the nodes of the tree for each row of data interpretation, which would introduce a large number of branches and virtual method calls.

def compile(node: Node): AST = node match {  caseq"$value"  caseq"row.get($name)"  caseq"${compile(left)} + ${compile(right)}"}

This part of the code is located org.spark.sql.catalyst.expressions.codegen.CodeGenerator.scala .

Catalyst Optimizer Optimizer