Use tornado coroutine for programming

After tornado's coroutine was released, the coroutine concept was enhanced. in asynchronous programming, the original gen. engine was replaced and changed to the current gen. coroutine. The decorator was originally designed to simplify asynchronous programming in tornado. Avoid writing callback functions to make development more in line with normal logic thinking. A simple example is as follows:

Class MaindHandler (web. RequestHandler ):

@ Asynchronous

@ Gen. coroutine

Def post (self ):

Client = AsyncHTTPClient ()

Resp = yield client. fetch (https://api.github.com/users ")

If resp. code = 200:

Resp = escape. json_decode (resp. body)

Self. write (json. dumps (resp, indent = 4, separators = (',',':')))

Else:

Resp = {"message": "error when fetch something "}

Self. write (json. dumps (resp, indent = 4, separators = {',',':')))

Self. finish ()

After the yield statement is run, ioloop registers the event and continues execution after the resp returns. This process is asynchronous. Here, json. dumps is used instead of tornado's own escape. json_encode, because JSON-format data is often accessed from a browser when constructing a REST-style API. After formatting data using json. dumps, it is more friendly to display and view data in the browser. Github API is the user of this style. In fact, escape. json_encode is for json. the simple packaging of dumps. when I put forward the pull request to package more functions, the author's answer to escape does not intend to provide all the json Functions. users can directly use the json module by themselves.

Principle of Gen. coroutine

As mentioned in the previous blog, to use the asynchronous feature of tornado, you must use an asynchronous Library. Otherwise, a single process will be blocked and will not achieve the asynchronous effect at all. The most common asynchronous library of Tornado is the built-in AsyncHTTPClient and the OpenID logon verification interface based on it. More asynchronous libraries can be found here. Including the use of a large number of MongoDB Driver.

After version 3.0, the gen. coroutine module became prominent. The coroutine decorator can make asynchronous programming based on callback look like Synchronous programming. The Send function of the generator in Python is used. In the generator, the yield keyword is often compared with the return in the normal function. It can be used as an iterator to return yield results using next. However, another usage of the generator is to use the send method. In the generator, yield results can be assigned to a variable, which is sent by the external generator client. For example:

Def test_yield ():

Pirnt "test yeild"

Says = (yield)

Print says

If _ name _ = "_ main __":

Client = test_yield ()

Client. next ()

Client. send ("hello world ")

The output result is as follows:

Test yeild

Hello world

A function that is already running will be suspended until the client that calls it uses the send method. the original function continues to run. Here gen. the coroutine method is used to asynchronously execute the required operations. after the result is returned, it is sent to the original function. the original function will continue to be executed, in this way, the code written in synchronous mode achieves the asynchronous execution effect.

Tornado asynchronous programming

Use coroutine to implement asynchronous programming of function separation. The details are as follows:

@ Gen. coroutine

Def post (self ):

Client = AsyncHTTPClient ()

Resp = yield client. fetch ("https://api.github.com/users ")

If resp = 200:

Body = escape. json_decode (resy. body)

Else:

Body = {"message": "client fetch error "}

Logger. error ("client fetch error % d, % s" % (resp. code, resp. message ))

Self. write (escape. json_encode (body ))

Self. finish ()

After changing to a function, it can become like this;

@ Gen. coroutime

Def post (self ):

Resp = yield GetUser ()

Self. write (resp)

@ Gen. coroutine

Def GetUser ():

Client = AsyncHTTPClient ()

Resp = yield client. fetch ("https://api.github.com/users ")

If resp. code = 200:

Resp = escape. json_decode (resp. body)

Else:

Resp = {"message": "fetch client error "}

Logger. error ("client fetch error % d, % s" % (resp. code, resp. message ))

Raise gen. Return (resp)

Here, when an asynchronous function is encapsulated, the return keyword is not used as a normal program. The gen module provides a gen. Return method. It is implemented through the raise method. This is also related to the generator implementation.

Use coroutine to run scheduled tasks

Tornado has the following method:

Tornado. ioloop. IOLoop. instance (). add_timeout ()

This method is a non-blocking version of time. sleep. it accepts two parameters: a time length and a function. Indicates the time after which the function is called. Here it is based on ioloop, so it is non-blocking. This method is used in client persistent connections and callback function programming. However, using it to run some scheduled tasks is helpless. It is usually unnecessary to run a scheduled task. However, when using heroku, I found that if I didn't register a credit card, I could only use a simple Web Application for hosting. You cannot add scheduled tasks to run tasks. So I came up with this method. Here, I mainly use it to capture data through the Github API interface every other time. You can use maxcompute as follows:

Decorator

Def sync_loop_call (delta = 60*1000 ):

"""

Wait for func down then process add_timeout

"""

Def wrap_loop (func ):

@ Wraps (func)

@ Gen. coroutine

Def wrap_func (* args, ** kwargs ):

Options.logger.info ("function % r start at % d" %

(Func. _ name __, int (time. time ())))

Try:

Yield func (* args, ** kwargs)

Except t Exception, e:

Options. logger. error ("function % r error: % s" %

(Func. _ name __, e ))

Options.logger.info ("function % r end at % d" %

(Func. _ name __, int (time. time ())))

Tornado. ioloop. IOLoop. instance (). add_timeout (

Datetime. timedelta (milliseconds = delta ),

Wrap_func)

Return wrap_func

Return wrap_loop

Task functions

@ Sync_loop_call (delta = 10*1000)

Def worker ():

"""

Do something

"""

Add task

If _ name _ = "_ main __":

Worker ()

App. listen (options. port)

Tornado. ioloop. IOLoop. instance (). start ()

After this is done, after the Web Application is started, the scheduled task will run as it is based on events and is executed asynchronously, therefore, it does not affect the normal operation of Web services. of course, tasks cannot be congested or computing-intensive. I mainly capture data here, and use the asynchronous crawling method that comes with Tornado.

In the sync_loop_call modifier, I added the @ gen. coroutine modifier to the wrap_func function. This ensures that the add_timeout operation will be executed only after the yeild function is executed. If there is no @ gen. coroutine modifier. Before yeild returns, add_timeout is executed.

For the complete example, see my Github. this project is built on heroku. Displays the ranking of Github user activity and regional user distribution. Visit Github-Data to view details. Because heroku in China is walled, you need to flip the wall to access it.

Summary

Tornado is a non-blocking web server and web framework. However, when used, only the asynchronous library can truly leverage its advantages, of course, sometimes because the App itself is not very demanding, if it is not blocked, it will not be a problem. In addition, when the coroutine module is used for asynchronous programming, when a function is encapsulated into a function, even if an error occurs during function running, it will not be thrown if it is not captured, this is very difficult for debugging.