Analysis of core technical points of Uber Athenax project __athenax

This paper analyzes the core technical points of Uber Athenax project. Overview

The following section is excerpted from the official document of the project.

Athenax is a streaming analysis platform that allows users to run SQL for large-scale, scalable streaming analysis. From Uber open source, with the expansion to hundreds of nodes to deal with daily Chi other real-time events.

The architecture diagram is as follows:

Technical notes built on Apache calcite and Apache Flink, yarn cluster to manage job leveldb as persistent storage Features streaming SQL
filtering, projecting and combining streams Aggregation on group windows over both processing and event time user-defined Functions (UDF), user-defined aggregation function (UDAF), and user-defined table functions (UDTF) (coming soon) efficient Executions through optimizations and code generations mechanisms to automatically fail over across multiple data centers Auto scaling for Athenax jobs Core Technical point athenax-backend

The back-end service implementation of the project, which provides a run-time instance. The main startup steps are divided into two steps: launching a Web server to receive various service requests from the restful;

The Web server here, in fact a lightweight HTTP server provided by the grizzly (server based on Java NIO) in a glashfish (Java EE Application Server implementation), also has the ability to handle dynamic requests (web Container,servlet) ability.

The Web server receives the user's RESTful API request, which can be grouped into three categories:

(1) Cluster: cluster-related information;
(2) Instance:job related information;
(3) Job: The information of the job itself;

RESTful API this block, Athenax uses the current more popular Swagger API development framework to provide part of the code (entity class/Service interface Class) generation. Starts a server's context, which encapsulates some of the core objects that are specific to the service provider:
Job Store: An opportunity leveldb job metadata storage mechanism; Job Manager: Note that this has nothing to do with Flink's JobManager, an object encapsulated in Athenax that manages the SQL job; instance Manager: A instance manager manages all the jobs that are being executed on the yarn cluster, watch Dog: Provides the status of the job, heartbeat detection, in a timely manner failover; Athenax-vm-compiler

Three component:planer: Scheduler, the entrance of the module, it will call parser, validator, executor in sequence, and finally get a Jobcompilationresult object called the result of the job compilation; Parser: Compiler, here is mainly for its expansion of the SQL to provide the corresponding analytical implementation, mainly to the implementation of the calcite API, and finally get Sqlnode collection sqlnodelist; Executor: Really complete the so-called "compile" work, Here compiled quotes, in fact, only with the help of Flink API to get the corresponding jobgraph;

Here, it is worth mentioning the "compilation" of the implementation mechanism. Athenax is ultimately going to commit its SQL job to the Flink runtime, and for Flink Jobgraph is the object of its uniquely identified job description, so the key point is to get the jobgraph of its job. So how does it do that. generation of Jobgraph

It (Jobcompiler) mocks a table&sql program template written using the Flink Table&sql API:

Streamexecutionenvironment execenv = Streamexecutionenvironment.createlocalenvironment ();
Streamtableenvironment env = streamtableenvironment.gettableenvironment (execenv);
Execenv.setstreamtimecharacteristic (timecharacteristic.eventtime);
Compilationresult res = new Compilationresult ();

try {
    Jobdescriptor job = getjobconf (system.in);
    Res.jobgraph (New Jobcompiler (env, Job). Getjobgraph ());
catch (Throwable e) {
    res.remotethrowable (e);
}

The core is the Getjobgraph method above

Jobdescriptor is the information about its job business, and then dynamically sets the unfixed part: input catalog:table source Udf:user defined Table/scalar/agg function sql:business SQL Output Catalog:sink

  Jobgraph Getjobgraph () throws IOException {streamexecutionenvironment exeenv
    = env.execenv ();
    Exeenv.setparallelism (Job.parallelism ());
    This
        . Registerudfs ()
        . Registerinputcatalogs ();
    Table table = Env.sql (Job.sql ());
    For (String t:job.outputs (). Listtables ()) {
      Table.writetosink (getoutputtable (Job.outputs (). GetTable (t)));
    streamgraph streamgraph = Exeenv.getstreamgraph ();
    return Streamgraph.getjobgraph ();
  }

Where the call to Env.sql () means that it does not really break away from Flink Table&sql

After the setup is complete, the Jobgraph object can be automatically obtained by calling Streamexecutionenvironment#getstreamgraph, so the Jobgraph generation is provided by Flink itself, and Athenax only needs piecing together and triggers the generation of the object.

After the build, the Jobgraph is submitted to the yarn cluster through the Flink yarn client implementation, and the Flink runtime execution job is started.

and the specific trigger mechanism, here Athenx adopted the run-time execution of the construction command line to execute the Jobcompiler method, and then use the socket + standard output Redirect to simulate the Unix PIPELINE, in fact, personally think that there is no need for such a detour, direct call on the line. code generation for parsers

It is worth mentioning that parser involves specific syntax, which is intended to embody flexibility. Athenax binds the parser's implementation class to SQL syntax to generate code in the form of a fmpp (text template preprocessor).

FMPP is a text preprocessor that supports Freemark syntax. Athenax-vm-api

This module is Athenax provided to the user to implement some of the API interface, they are: function: the rich of various functions (Open/close method) extension, catalog:table/source, sink mapping; Provider:sink extension interface; athennax-vm-connectors

Open to the user to extend the connector, currently only provides the Kafka this one connector implementation. Summary

Athenax code is not large and complex, but it provides a mechanism for extending the flink to take advantage of its runtime.