Introduced
One of the biggest problems with using Python for web development is performance, which is a bit hard to solve c10k problems. Some asynchronous frameworks tornado, Twisted, gevent, etc. are designed to solve performance problems. These frameworks are somewhat elevated in performance, but there are a variety of bizarre problems that are difficult to solve.
In python3.6, the official Asynchronous association threading Asyncio formally become the standard. While retention convenience has greatly improved performance, there have been many asynchronous frameworks using Asyncio.
Using an earlier asynchronous framework is Aiohttp, which provides a server-side and client-side, and is a good encapsulation of Asyncio. However, the development method and the most popular micro-frame flask different, flask development is simple, lightweight, efficient.
Micro-service is the most recent development mode, it solves the complexity problem, improve the development efficiency, easy to deploy and other advantages.
It is the combination of these advantages, based on sanic, the integration of a number of popular libraries to build micro-services. The Sanic framework is a similar asynchronous flask framework, simple and lightweight, and has a high performance.
This project is a micro-service framework based on Sanic.
Characteristics
Use Sanic asynchronous framework, simple, lightweight, and efficient.
The use of Uvloop as the core engine, so that sanic in many cases of stand-alone concurrent even as Golang.
Use ASYNCPG for database-driven, database connections, execution of SQL statements.
Use Aiohttp as client to access other micro services.
Use PeeWee for ORM, but only for model design and migration.
Use Opentracing as a distributed tracking system.
Use UnitTest to do unit tests and use mocks to avoid accessing other micro services.
API documentation can be generated automatically using the Swagger API standard.
Use
Project Address
Sanic-ms: ' Https://github.com/songcser/sanic-ms '
Example: ' Https://github.com/songcser/sanic-ms/tree/master/examples '
Swagger API
Zipkin Server
Service side
With the Sanic asynchronous framework, there is high performance, but improper use will cause blocking, for the IO request to choose the asynchronous library. It is prudent to add libraries. Sanic uses uvloop asynchronous drive, Uvloop is based on LIBUV to write with Cython, performance is higher than Nodejs.
Function description
Before starting
@app. Listener (' Before_server_start ')
Async def before_srver_start (app, loop):
Queue = Asyncio. Queue ()
App.queue = Queue
Loop.create_task (Consume (queue, App.config.ZIPKIN_SERVER))
Reporter = Aioreporter (queue=queue)
Tracer = Basictracer (Recorder=reporter)
Tracer.register_required_propagators ()
Opentracing.tracer = Tracer
App.db = await ConnectionPool (loop=loop). Init (Db_config)
Create DB connection Pool
Create a Client connection
Create a queue that consumes span for log tracking
Create opentracing.tracer for log tracking
Middleware
@app. Middleware (' request ')
Async def Cros (request):
if Request.method = = ' POST ' or Request.method = ' put ':
request[' data '] = Request.json
span = before_request (Request)
Request[' span '] = span
@app. Middleware (' response ')
Async def cors_res (Request, Response):
span = request[' span '] if ' span ' in request else None
If response is None:
return response
result = {' Code ': 0}
If not isinstance (response, HttpResponse):
If Isinstance (response, tuple) and len (response) = 2:
Result.update ({
' Data ': response[0],
' Pagination ': response[1]
})
Else
Result.update ({' Data ': response})
Response = JSON (Result)
If span:
Span.set_tag (' Http.status_code ', "200")
If span:
Span.set_tag (' component ', request.app.name)
Span.finish ()
return response
Create span for log tracking
Encapsulation of response, uniform format
Exception handling
Handles the thrown exception, returning to the uniform format
Task
Create a task consumption queue for span, for log tracking
Asynchronous processing
Because the asynchronous framework is used, some IO requests can be processed in parallel
Example:
Async def async_request (datas):
# Async Handler Request
Results = await asyncio.gather (*[data[2] for data in datas])
For index, obj in enumerate (results):
data = Datas[index]
DATA[0][DATA[1]] = Results[index]
@user_bp. Get ('/<id:int> ')
@doc. Summary ("Get user info")
@doc. Description ("Get user info by id")
@doc. produces (Users)
Async def get_users_list (request, ID):
Async with Request.app.db.acquire (request) as cur:
Record = await Cur.fetch (
"" "Select * from users WHERE id =" ", id)
Datas = [
[Record, ' city_id ', get_city_by_id (Request, record[' city_id ']]
[Record, ' role_id ', get_role_by_id (Request, record[' role_id ']]
]
Await Async_request (datas)
return record
Getcitybyid, Getrolebyid is parallel processing.
Related connections
Sanic:https://github.com/channelcat/sanic
Model Design & ORM
PeeWee is a simple and small ORM. It has few (but expressive) concepts, making it easy to learn and intuitive to use.
ORM uses peewee, just to do model design and migration, database operations using ASYNCPG.
Example:
# models.py
Class Users (Model):
id = Primarykeyfield ()
Create_time = Datetimefield (verbose_name= ' Create Time ',
Default=datetime.datetime.utcnow)
Name = Charfield (max_length=128, verbose_name= "user ' name")
Age = Integerfield (Null=false, verbose_name= "user's Age")
Sex = Charfield (max_length=32, verbose_name= "user ' s Sex")
city_id = Integerfield (verbose_name= ' City for user ', HELP_TEXT=CITYAPI)
role_id = Integerfield (verbose_name= ' role for user ', HELP_TEXT=ROLEAPI)
Class Meta:
db_table = ' users '
# migrations.py
From sanic_ms.migrations import Migrationmodel, info, db
Class Usermigration (Migrationmodel):
_model = Users
# @info (version= "v1")
# def MIGRATE_V1 (self):
# Migrate (self.add_column (' sex '))
def migrations ():
Try
Um = Usermigration ()
With Db.transaction ():
Um.auto_migrate ()
Print ("Success migration")
Except Exception as E:
Raise E
if __name__ = = ' __main__ ':
Migrations ()
Run command python migrations.py
MIGRATE_V1 function to add a field sex, in Basemodel to add the Name field first
The info adorner creates a table Migrate_record to record migrate,version each model must be unique, uses version to record whether it has been executed, and can also record author,datetime
Migrate function must begin with Migrate_
Related connections
peewee:http://docs.peewee-orm.com/en/latest/
Database operations
ASYNCPG is the fastest driver among common Python, Nodejs and go implementations
Use ASYNCPG for database drive, encapsulate database connections, perform database operations.
Do not use ORM to do database operations, one reason is performance, ORM will have performance loss, and can not use the ASYNCPG high-performance library. Another is that a single micro-service is simple, the table structure is not very complex, simple SQL statements can be processed, there is no need to introduce ORM. Using peewee just to do model design
Example:
sql = "SELECT * from users WHERE name=$1"
Name = "Test"
Async with Request.app.db.acquire (request) as cur:
data = await cur.fetchrow (SQL, name)
Async with Request.app.db.transaction (request) as cur:
data = await cur.fetchrow (SQL, name)
The acquire () function is not a transaction, and can improve query efficiency for the use of non transactions only involving queries
The Tansaction () function is a transaction operation, and a transaction operation must be used for additions and deletions
Incoming request parameter is to get span, for log tracking
TODO database read-write separation
Related connections
Asyncpg:https://github.com/magicstack/asyncpg
benchmarks:https://magic.io/blog/asyncpg-1m-rows-from-postgres-to-python/
Client
Using the client in Aiohttp, a simple encapsulation of clients is performed for access between micro services.
Don ' t create a session per request. Most likely you need a sessions per application which performs all requests altogether.
A session contains a connection pool inside, connection Reusage and Keep-alives (both are on by default) may speed up Tota L performance.
Example:
@app. Listener (' Before_server_start ')
Async def before_srver_start (app, loop):
app.client = Client (loop, url= ' Http://host:port ')
Async def get_role_by_id (request, ID):
CLI = REQUEST.APP.CLIENT.CLI (Request)
Async with Cli.get ('/cities/{} '. Format (ID)) as res:
return await Res.json ()
@app. Listener (' Before_server_stop ')
Async def before_server_stop (app, loop):
App.client.close ()
For access to different micro services can create multiple different client, so that each client will keep-alives
Related connections
Aiohttp:http://aiohttp.readthedocs.io/en/stable/client.html
Log & Distributed Tracking System
Use the official logging, the configuration file is Logging.yml, sanic version to 0.6.0 and above. Jsonformatter log into JSON format for input to ES
Enter Opentracing:by offering consistent, expressive, vendor-neutral APIs for popular platforms, Opentracing makes it eas Y for developers to add (or switch) tracing implementations with a O (1) configuration change. Opentracing also offers a lingua franca for OSS instrumentation and platform-specific helper tracing. Please refer to the semantic specification.
Adorner Logger
@logger (type= ' method ', category= ' test ', detail= ' detail ', description= "des", tracing=true, Level=logging.info)
Async def get_city_by_id (request, ID):
CLI = REQUEST.APP.CLIENT.CLI (Request)
Type: Log types, such as method, route
Category: Log category, default to App name
Detail: Log Details
Description: Log description, default to function annotation
Tracing: Log tracking, defaults to True
Level: Log levels, default to info
Distributed Tracking System
Opentracing is based on the distributed tracking system such as Dapper,zipkin and establishes a unified standard.
Opentracing tracking each request, recording the request through each of the micro-service, chain-by-line, the analysis of micro-service performance bottlenecks is critical.
Use the opentracing framework, but convert to Zipkin format at output time. Because most distributed tracking systems take into account performance issues, all are used to communicate with thrift, in the spirit of simplicity, restful style, without RPC communication. In the form of log output, you can use Fluentd, Logstash, and other log collection and then input to Zipkin. Zipkin is supported for HTTP input.
The generated span is first placed in the queue without blocking, and spans in the task's consumption queues. The late sampling frequency can be added later.
For Db,client, the tracing is added.
Related connections
Opentracing:https://github.com/opentracing/opentracing-python
Zipkin:https://github.com/openzipkin/zipkin
jaeger:https://uber.github.io/jaeger/
API interface
API documentation uses the Swagger standard.
Example:
From Sanic_ms import doc
@user_bp. Post ('/')
@doc. Summary (' Create user ')
@doc. Description (' Create user info ')
@doc. Consumes (Users)
@doc. produces ({' id ': int})
Async def create_user (request):
data = request[' data ']
Async with Request.app.db.transaction (request) as cur:
Record = await Cur.fetchrow (
"" INSERT into the users (name, age, city_id, role_id)
VALUES ($, $, $, $ $)
Returning ID
"" ", data[' name '], data[' age ', data[' city_id ', data[' role_id ']
)
return {' id ': record[' id ']}
SUMMARY:API Overview
Description: Detailed description
Body Data for Consumes:request
Produces:response's return Data
Tag:api Label
The parameters passed in consumes and produces can be the model of PeeWee, parsing model generates API data, and the Help_text parameters in field fields represent reference objects
Http://host:ip/openapi/spec.json gets the generated JSON data
Related connections
swagger:https://swagger.io/
Response data
In return, do not return sanic response, directly return the original data, will be in the middleware of the returned data processing, return a uniform format, the specific format can [view]
Unit Test
Unit tests use UnitTest. A mock is a mockclient created by itself, because UnitTest has no Asyncio mocks, and the Sanic test interface sends request requests, so it's more cumbersome. Pytest can be used later.
Example:
From sanic_ms.tests import apitestcase
From server Import app
Class TestCase (Apitestcase):
_app = App
_blueprint = ' Visit '
def setUp (self):
Super (TestCase, self). SetUp ()
Self._mock.get ('/cities/1 ',
payload={' id ': 1, ' name ': ' Shanghai '}
Self._mock.get ('/roles/1 ',
payload={' id ': 1, ' name ': ' Shanghai '}
def test_create_user (self):
data = {
' Name ': ' Test ',
' Age ': 2,
' city_id ': 1,
' role_id ': 1,
}
res = Self.client.create_user (data=data)
BODY = ujson.loads (Res.text)
Self.assertequal (Res.status, 200)
Where _blueprint is the blueprint name
In the Setup function, use _mock to register mock information so that the real server is not accessed, payload for the returned body information
Use the client variable to call each function, data is the body information, params is the parameter information of the path, the other parameters are route parameters
Code overwrite
Coverage Erase
Coverage run--source. -M Sanic_ms tests
Coverage Xml-o Reports/coverage.xml
Coverage2clover-i Reports/coverage.xml-o Reports/clover.xml
Coverage html-d reports