Asynchronous programming using Tornado's Gen.coroutine

After the release of Tornado3, the concept of coroutine was strengthened, and in asynchronous programming, it replaced the original Gen.engine and became the present gen.coroutine. This decorator is intended to simplify asynchronous programming in tornado. Avoid writing callback functions that make development more consistent with normal logical thinking.
A simple example is as follows:

2. Class Maindhandler(Web. RequestHandler):
4. @asynchronous
6. @gen. Coroutine
8. def Post(self):
10. Client = asynchttpclient()
12. Resp = yield client. Fetch(HTTPS://API. GitHub. COM/users")
14. if Resp.code = = 200:
16. RESP = Escape.json_decode (resp.body)
18. Self.write (Json.dumps (resp, indent=4, separators= (', ', ': ')))
20. Else
22. Resp = {"message": "error when fetch something"}
24. Self.write (Json.dumps (resp, indent=4, separators={', ', ': ')))
26. Self.finish ()

After the yield statement, Ioloop registers the event until RESP returns and resumes execution. This process is asynchronous. Using Json.dumps here instead of using Tornado's escape.json_encode is because when building a restful API, it is often accessed from the browser to get JSON-formatted data. After you format the data using Json.dumps, it is more friendly when the browser side displays the view. The Github API is the user of this style. In fact, Escape.json_encode is a simple package for json.dumps, and when I ask for more versatility with the pull request package, the author's answer to escape is not intended to provide full JSON functionality, and the user can use the JSON module directly.

gen.coroutine principle

In a previous blog post about the asynchronous nature of using tornado, you must use an asynchronous library. Otherwise, a single process is blocked and does not achieve an asynchronous effect at all. The most common in Tornado asynchronous libraries is the self-contained asynchttpclient, and the OpenID login authentication interface implemented on top of it. Additional asynchronous libraries can be found here. Including the use of more MongoDB driver.

After the 3.0 release, the Gen.coroutine module appears to be more prominent. The Coroutine decorator allows asynchronous programming that would otherwise be callbacks to look like synchronous programming. This is the use of the Send function of the generator in Python. In the generator, the yield keyword tends to be compared to return in the normal function. It can be used as an iterator to return yield results using next (). But there is another use of the generator, which is using the Send method. The result of yield can be assigned to a variable inside the generator, and this value is sent through the external generator client. To give an example:

2. def Test_yield():
4. Pirnt "Test Yeild"
6. Says = (yield)
8. print says
12. If __name__ = = "__main__":
14. Client = Test_yield()
16. Client. Next()
18. Client. Send("Hello World")

The output results are as follows:
Test Yeild
Hello World

The function that is already running is suspended until the client calling it uses the Send method, and the original function continues to run. The Gen.coroutine method here is to execute the required operation asynchronously, then wait for the result to return, then send to the original function, the original function will continue to execute, so that the synchronous writing code to achieve the effect of asynchronous execution.

Tornado Asynchronous Programming

Asynchronous programming that uses Coroutine to implement function separation. Specific as follows:

2. @gen. Coroutine
4. def Post(self):
6. Client = asynchttpclient()
8. Resp = yield client. Fetch("Https://api.github.com/users")
10. if resp = = :
12. Body = escape. Json_decode(resy. Body)
14. else:
16. Body = {"message": "client fetch error"}
18. Logger. Error("Client fetch error%d,%s" % (resp). Code, resp. Message))
20. Self. Write(escape. Json_encode(body))
22. Self. Finish()

This can be changed after the function is changed;

2. @gen. Coroutime
4. def Post(self):
6. Resp = yield GetUser()
8. Self. Write(resp)
12. @gen. Coroutine
14. def GetUser():
16. Client = asynchttpclient()
18. Resp = yield client. Fetch("Https://api.github.com/users")
20. if resp. Code = = :
22. Resp = escape. Json_decode(resp. Body)
24. else:
26. Resp = {"message": "Fetch client Error"}
28. Logger. Error("Client fetch error%d,%s" % (resp). Code, resp. Message))
30. raise gen. Return(resp)

Here, when the asynchronous encapsulated in a function, not like a normal program with the return keyword to return, Gen module provides a Gen. The return method. is achieved through the Raise method. This is also related to the way it is implemented using the generator.

use Coroutine to run timed tasks

There is one such method in tornado:

Tornado.ioloop.IOLoop.instance (). Add_timeout ()

The method is a non-blocking version of Time.sleep, which accepts two parameters: a length of time and a function. Represents how much time is called after the function. Here it is based on ioloop, so it is non-blocking. This method is more used in long-client connection and callback function programming. But using it to run some timed tasks is helpless. Usually run-time tasks are also not necessary to use to it. But when I was using Heroku, I found that I could only use a simple Web application hosting without registering a credit card. You cannot add timed tasks to run. So I came up with such a method. Here, I mostly use it to crawl data over a period of time through the GitHub API interface. The large self-use method is as follows:

Decorative Device

2. def sync_loop_call(delta= * ):
4. """
6. Wait for func down then process add_timeout
8. """
10. def wrap_loop(func):
12. @wraps(func)
14. @gen. Coroutine
16. def wrap_func(*args, * *Kwargs):
18. Options. Logger. Info("function%r start at%d" %
20. (func. __name__, int(time. Time() )))
22. try:
24. yield func(*args, * *kwargs)
26. except Exception, e:
28. Options. Logger. Error("function%r Error:%s" %
30. (func. __name__, e))
32. Options. Logger. Info("function%r end at%d" %
34. (func. __name__, int(time. Time() )))
36. Tornado. Ioloop. Ioloop. Instance(). Add_timeout(
38. DateTime. Timedelta(milliseconds=Delta),
40. Wrap_func)
42. return wrap_func
44. return wrap_loop

Task Function

1. @sync_loop_call(delta=Ten * )
2. def worker():
3. """
4. Do something
5. """

Add a task

1. if __name__ = = "__main__":
2. Worker()
3. App. Listen(options. Port)
4. Tornado. Ioloop. Ioloop. Instance(). Start()

After doing this, when the Web application is started, the scheduled task will run with it, and because it is event-based and asynchronous, it does not affect the normal operation of the Web service, and the task must not be blocked or computationally intensive. I'm mainly crawling data here, and using Tornado's own asynchronous fetch method.

In the Sync_loop_call decorator, I added a @gen.coroutine adorner on the Wrap_func function, which ensures that only the Yeild function is executed before the add_timeout operation is performed. If there is no @gen.coroutine adorner. Then the add_timeout will be executed without waiting for Yeild to return.

A complete example can be found on my GitHub, a project built on Heroku. Used to show GitHub user activity rankings and user area distributions. You can access the Github-data view. As the domestic Heroku by the wall, need to FQ to access.

Summary

Tornado is a non-blocking Web server as well as a web framework, but only using an asynchronous library in use can really play its asynchronous advantage, sometimes because the app itself is not very demanding, and if it is not blocking is particularly serious, there is no problem. In addition to using the Coroutine module for asynchronous programming, when a function is encapsulated in a function, in the function, even if there is an error, if not to capture the words will not be thrown, which is very difficult to debug.

Asynchronous programming using Tornado's Gen.coroutine