Sanic based micro-service infrastructure

Introduced

One of the biggest problems with using Python for web development is performance, which is a bit hard to solve c10k problems. Some asynchronous frameworks tornado, Twisted, gevent, etc. are designed to solve performance problems. These frameworks are somewhat elevated in performance, but there are a variety of bizarre problems that are difficult to solve.

In python3.6, the official Asynchronous association threading Asyncio formally become the standard. While retention convenience has greatly improved performance, there have been many asynchronous frameworks using Asyncio.

Using an earlier asynchronous framework is Aiohttp, which provides a server-side and client-side, and is a good encapsulation of Asyncio. However, the development method and the most popular micro-frame flask different, flask development is simple, lightweight, efficient.

Micro-service is the most recent development mode, it solves the complexity problem, improve the development efficiency, easy to deploy and other advantages.

It is the combination of these advantages, based on sanic, the integration of a number of popular libraries to build micro-services. The Sanic framework is a similar asynchronous flask framework, simple and lightweight, and has a high performance.

This project is a micro-service framework based on Sanic.

Characteristics

Use Sanic asynchronous framework, simple, lightweight, and efficient.

The use of Uvloop as the core engine, so that sanic in many cases of stand-alone concurrent even as Golang.

Use ASYNCPG for database-driven, database connections, execution of SQL statements.

Use Aiohttp as client to access other micro services.

Use PeeWee for ORM, but only for model design and migration.

Use Opentracing as a distributed tracking system.

Use UnitTest to do unit tests and use mocks to avoid accessing other micro services.

API documentation can be generated automatically using the Swagger API standard.

Use

Project Address

     
     
      
      
Sanic-ms: ' Https://github.com/songcser/sanic-ms '
      
      

      
      
Example: ' Https://github.com/songcser/sanic-ms/tree/master/examples '
     
     

Swagger API

Zipkin Server

Service side

With the Sanic asynchronous framework, there is high performance, but improper use will cause blocking, for the IO request to choose the asynchronous library. It is prudent to add libraries. Sanic uses uvloop asynchronous drive, Uvloop is based on LIBUV to write with Cython, performance is higher than Nodejs.

Function description

Before starting

     
     
      
      
@app. Listener (' Before_server_start ')
      
      
Async def before_srver_start (app, loop):
      
      
Queue = Asyncio. Queue ()
      
      
App.queue = Queue
      
      
Loop.create_task (Consume (queue, App.config.ZIPKIN_SERVER))
      
      
Reporter = Aioreporter (queue=queue)
      
      
Tracer = Basictracer (Recorder=reporter)
      
      
Tracer.register_required_propagators ()
      
      
Opentracing.tracer = Tracer
      
      
App.db = await ConnectionPool (loop=loop). Init (Db_config)
     
     

Create DB connection Pool

Create a Client connection

Create a queue that consumes span for log tracking

Create opentracing.tracer for log tracking

Middleware

     
     
      
      
@app. Middleware (' request ')
Async def Cros (request):
if Request.method = = ' POST ' or Request.method = ' put ':
request[' data '] = Request.json
span = before_request (Request)
Request[' span '] = span
@app. Middleware (' response ')
Async def cors_res (Request, Response):
span = request[' span '] if ' span ' in request else None
If response is None:
return response
result = {' Code ': 0}
If not isinstance (response, HttpResponse):
If Isinstance (response, tuple) and len (response) = 2:
Result.update ({
' Data ': response[0],
' Pagination ': response[1]
})
Else
Result.update ({' Data ': response})
Response = JSON (Result)
If span:
Span.set_tag (' Http.status_code ', "200")
If span:
Span.set_tag (' component ', request.app.name)
Span.finish ()
return response

Create span for log tracking

Encapsulation of response, uniform format

Exception handling

Handles the thrown exception, returning to the uniform format

Task

Create a task consumption queue for span, for log tracking

Asynchronous processing

Because the asynchronous framework is used, some IO requests can be processed in parallel

Example:

     
     
      
      
Async def async_request (datas):
# Async Handler Request
Results = await asyncio.gather (*[data[2] for data in datas])
For index, obj in enumerate (results):
data = Datas[index]
DATA[0][DATA[1]] = Results[index]
@user_bp. Get ('/<id:int> ')
@doc. Summary ("Get user info")
@doc. Description ("Get user info by id")
@doc. produces (Users)
Async def get_users_list (request, ID):
Async with Request.app.db.acquire (request) as cur:
Record = await Cur.fetch (
"" "Select * from users WHERE id =" ", id)
Datas = [
[Record, ' city_id ', get_city_by_id (Request, record[' city_id ']]
[Record, ' role_id ', get_role_by_id (Request, record[' role_id ']]
]
Await Async_request (datas)
return record

Getcitybyid, Getrolebyid is parallel processing.

Related connections

Sanic:https://github.com/channelcat/sanic

Model Design & ORM

     
     
      
      
PeeWee is a simple and small ORM. It has few (but expressive) concepts, making it easy to learn and intuitive to use.
     
     

ORM uses peewee, just to do model design and migration, database operations using ASYNCPG.

Example:

     
     
      
      
# models.py
Class Users (Model):
id = Primarykeyfield ()
Create_time = Datetimefield (verbose_name= ' Create Time ',
Default=datetime.datetime.utcnow)
Name = Charfield (max_length=128, verbose_name= "user ' name")
Age = Integerfield (Null=false, verbose_name= "user's Age")
Sex = Charfield (max_length=32, verbose_name= "user ' s Sex")
city_id = Integerfield (verbose_name= ' City for user ', HELP_TEXT=CITYAPI)
role_id = Integerfield (verbose_name= ' role for user ', HELP_TEXT=ROLEAPI)
Class Meta:
db_table = ' users '
# migrations.py
From sanic_ms.migrations import Migrationmodel, info, db
Class Usermigration (Migrationmodel):
_model = Users
# @info (version= "v1")
# def MIGRATE_V1 (self):
# Migrate (self.add_column (' sex '))
def migrations ():
Try
Um = Usermigration ()
With Db.transaction ():
Um.auto_migrate ()
Print ("Success migration")
Except Exception as E:
Raise E
if __name__ = = ' __main__ ':
Migrations ()

Run command python migrations.py

MIGRATE_V1 function to add a field sex, in Basemodel to add the Name field first

The info adorner creates a table Migrate_record to record migrate,version each model must be unique, uses version to record whether it has been executed, and can also record author,datetime

Migrate function must begin with Migrate_

Related connections

     
     
      
      
peewee:http://docs.peewee-orm.com/en/latest/
     
     

Database operations

     
     
      
      
ASYNCPG is the fastest driver among common Python, Nodejs and go implementations
     
     

Use ASYNCPG for database drive, encapsulate database connections, perform database operations.

Do not use ORM to do database operations, one reason is performance, ORM will have performance loss, and can not use the ASYNCPG high-performance library. Another is that a single micro-service is simple, the table structure is not very complex, simple SQL statements can be processed, there is no need to introduce ORM. Using peewee just to do model design

Example:

     
     
      
      
sql = "SELECT * from users WHERE name=$1"
      
      
Name = "Test"
      
      
Async with Request.app.db.acquire (request) as cur:
      
      
data = await cur.fetchrow (SQL, name)
      
      
Async with Request.app.db.transaction (request) as cur:
      
      
data = await cur.fetchrow (SQL, name)
     
     

The acquire () function is not a transaction, and can improve query efficiency for the use of non transactions only involving queries

The Tansaction () function is a transaction operation, and a transaction operation must be used for additions and deletions

Incoming request parameter is to get span, for log tracking

TODO database read-write separation

Related connections

     
     
      
      
Asyncpg:https://github.com/magicstack/asyncpg
      
      

      
      
benchmarks:https://magic.io/blog/asyncpg-1m-rows-from-postgres-to-python/
     
     

Client

Using the client in Aiohttp, a simple encapsulation of clients is performed for access between micro services.

     
     
      
      
Don ' t create a session per request. Most likely you need a sessions per application which performs all requests altogether.
      
      
A session contains a connection pool inside, connection Reusage and Keep-alives (both are on by default) may speed up Tota L performance.
     
     

Example:

     
     
      
      
@app. Listener (' Before_server_start ')
      
      
Async def before_srver_start (app, loop):
      
      
app.client = Client (loop, url= ' Http://host:port ')
      
      
Async def get_role_by_id (request, ID):
      
      
CLI = REQUEST.APP.CLIENT.CLI (Request)
      
      
Async with Cli.get ('/cities/{} '. Format (ID)) as res:
      
      
return await Res.json ()
      
      
@app. Listener (' Before_server_stop ')
      
      
Async def before_server_stop (app, loop):
      
      
App.client.close ()
     
     

For access to different micro services can create multiple different client, so that each client will keep-alives

Related connections

     
     
      
      
Aiohttp:http://aiohttp.readthedocs.io/en/stable/client.html
     
     

Log & Distributed Tracking System

Use the official logging, the configuration file is Logging.yml, sanic version to 0.6.0 and above. Jsonformatter log into JSON format for input to ES

     
     
      
      
Enter Opentracing:by offering consistent, expressive, vendor-neutral APIs for popular platforms, Opentracing makes it eas Y for developers to add (or switch) tracing implementations with a O (1) configuration change. Opentracing also offers a lingua franca for OSS instrumentation and platform-specific helper tracing. Please refer to the semantic specification.
     
     

Adorner Logger

     
     
      
      
@logger (type= ' method ', category= ' test ', detail= ' detail ', description= "des", tracing=true, Level=logging.info)
      
      
Async def get_city_by_id (request, ID):
      
      
CLI = REQUEST.APP.CLIENT.CLI (Request)
     
     

Type: Log types, such as method, route

Category: Log category, default to App name

Detail: Log Details

Description: Log description, default to function annotation

Tracing: Log tracking, defaults to True

Level: Log levels, default to info

Distributed Tracking System

Opentracing is based on the distributed tracking system such as Dapper,zipkin and establishes a unified standard.

Opentracing tracking each request, recording the request through each of the micro-service, chain-by-line, the analysis of micro-service performance bottlenecks is critical.

Use the opentracing framework, but convert to Zipkin format at output time. Because most distributed tracking systems take into account performance issues, all are used to communicate with thrift, in the spirit of simplicity, restful style, without RPC communication. In the form of log output, you can use Fluentd, Logstash, and other log collection and then input to Zipkin. Zipkin is supported for HTTP input.

The generated span is first placed in the queue without blocking, and spans in the task's consumption queues. The late sampling frequency can be added later.

For Db,client, the tracing is added.

Related connections

     
     
      
      
Opentracing:https://github.com/opentracing/opentracing-python
      
      

      
      
Zipkin:https://github.com/openzipkin/zipkin
      
      

      
      
jaeger:https://uber.github.io/jaeger/
     
     

API interface

API documentation uses the Swagger standard.

Example:

     
     
      
      
From Sanic_ms import doc
@user_bp. Post ('/')
@doc. Summary (' Create user ')
@doc. Description (' Create user info ')
@doc. Consumes (Users)
@doc. produces ({' id ': int})
Async def create_user (request):
data = request[' data ']
Async with Request.app.db.transaction (request) as cur:
Record = await Cur.fetchrow (
"" INSERT into the users (name, age, city_id, role_id)
VALUES ($, $, $, $ $)
Returning ID
"" ", data[' name '], data[' age ', data[' city_id ', data[' role_id ']
)
return {' id ': record[' id ']}

SUMMARY:API Overview

Description: Detailed description

Body Data for Consumes:request

Produces:response's return Data

Tag:api Label

The parameters passed in consumes and produces can be the model of PeeWee, parsing model generates API data, and the Help_text parameters in field fields represent reference objects

Http://host:ip/openapi/spec.json gets the generated JSON data

Related connections

     
     
      
      
swagger:https://swagger.io/
     
     

Response data

In return, do not return sanic response, directly return the original data, will be in the middleware of the returned data processing, return a uniform format, the specific format can [view]

Unit Test

Unit tests use UnitTest. A mock is a mockclient created by itself, because UnitTest has no Asyncio mocks, and the Sanic test interface sends request requests, so it's more cumbersome. Pytest can be used later.

Example:

     
     
      
      
From sanic_ms.tests import apitestcase
From server Import app
Class TestCase (Apitestcase):
_app = App
_blueprint = ' Visit '
def setUp (self):
Super (TestCase, self). SetUp ()
Self._mock.get ('/cities/1 ',
payload={' id ': 1, ' name ': ' Shanghai '}
Self._mock.get ('/roles/1 ',
payload={' id ': 1, ' name ': ' Shanghai '}
def test_create_user (self):
data = {
' Name ': ' Test ',
' Age ': 2,
' city_id ': 1,
' role_id ': 1,
}
res = Self.client.create_user (data=data)
BODY = ujson.loads (Res.text)
Self.assertequal (Res.status, 200)

Where _blueprint is the blueprint name

In the Setup function, use _mock to register mock information so that the real server is not accessed, payload for the returned body information

Use the client variable to call each function, data is the body information, params is the parameter information of the path, the other parameters are route parameters

Code overwrite

     
     
      
      
Coverage Erase
      
      
Coverage run--source. -M Sanic_ms tests
      
      
Coverage Xml-o Reports/coverage.xml
      
      
Coverage2clover-i Reports/coverage.xml-o Reports/clover.xml
      
      
Coverage html-d reports