Thesis excerpt-Tenzing

This excerpt does not guarantee the integrity and accuracy of the thesis.

Background

Heterogeneous Data adaptation and data scalability, resource scalability, cheap machines, and SQL queries.

Architecture Overview

Worker pool, query server, clientinterfaces, metadata server

The worker thread is long-running. The worker pool includes the master node and worker node, and a master watcher.

The query server parses the query and passes the optimization to the master for execution. The optimization includes some basic rule optimization and cost-based optimization.

SQL features

The SQL syntax is mainly sql92, and some advanced analysis is added. The enhanced part is also suitable for completely parallel mr.

The sawzall syntax is embedded and supports user-written and sawzall built-in methods.

 

In terms of ing and filtering,

Constant expressions are calculated in the compile stage;

If the predicate is a constant or the search is a constant range, you can directly push it to a data source such as bigtable;

If the predicate does not contain complex udfs and the data source is a database such as MySQL, it is also pushed down to the data source;

If it is partitioned, data in other partitions will be skipped;

Columns data out of the specified range is skipped based on the meta information of columnio;

If the following data source is column-oriented, Tenzing skips irrelevant columns.

 

In terms of aggregate functions,

Supports sum, Count, Min, Max, distinct, Count distinct,

Statistical-type Corr, covar, and stddev are also supported.

Some additional optimizations are added to the implementation, such as implementing hash table-based aggregation in mapreduce.

 

Join,

Tenzing supports various join methods across data sources.

Broadcast joins, the cost-based optimizer will find a small enough table to broadcast to the memory, so that Mapper and reducer can directly access the memory. There are several details to optimize.

Remote lookup joins. For example, if bigtable supports index-based Key search, it can be real-time.

Distributed sort-merge joins.

Distributer hash joins is suitable for situations where both tables cannot store memory and one table is much larger than the other, and the join key has no index. For more information about implementation and optimization, see the pseudocode and description in this article.

 

Analysis functions,

Similar to PostgreSQL/Oracle, it supports rank, sum, Min, Max, lead, lag, and ntile.

 

OLAP extension,

Supports rollup and cube.

 

Set Operations,

Supports standard SQL set operations, such as Union, Union all, minus, and minus all.

 

Nested query and subquery,

The number of Mr Jobs will be optimized.

 

Processing structured data,

According to the Pb protocol, some SQL statements do not support

 

Views,

Support for create views

 

DML,

The insert, update, and delete operations in batch mode are not acid-sensitive.

 

DDL,

Supports create table, drop table, rename table, generatestatistics, Grant, and revoke.

 

Table valued functions,

 

Data formats,

GFS, bigtable, columnio, etc.

Performance

The performance goal is to compare it with traditional MPP database systems.

 

Optimization and enhancement of mapreduce,

The work pool is designed to reduce latency, specifically masterwatcher, Master pool and worker pool.

Streaming & in-memory chaining, data convergence between Mrs is in the stream mode; the former Mapper and the latter reducer can be in the same process.

Avoid sort, hash join, hash aggregation, shuffle, and sort. Disable sort.

Block shuffle. The original row-based Shuffle is for sort. When sort is not required, the shuffle of about 1 MB is 3 times more efficient.

Local execution depends on the data volume (less than 128 MB.

Llvm Query Engine

A generation of engines translate SQL into sawzall code

The second generation engine uses dremel's SQL expression analysis engine.

Three generations of engines try to use llvm based row and vector based Column

I have pointed out some advantages and disadvantages and believe that the native codegeneration engine will be in the future.

Full Text :)