Learning and using the Python Asyncio Library

Because to find a job, the previous crawler to organize their own, no project experience real egg pain, can only do this kind of water is not the thing ... T T, hope to find a job can have good results.

Before the crawler used is requests+ multi-threaded/multi-process, and later with a few days in-depth understanding, only to find that, for the crawler, the real bottleneck is not the CPU processing speed, but for the page crawl time of the round trip, because if the use of requests+ multi-threaded/multi-process, He is blocking programming, so time is spent waiting for the return of the page results and for writing the crawled data. And if you use non-blocking programming, then there is no such problem. Here, first, we need to understand the difference between blocking and non-blocking.

1. A blocking call means that the current thread is suspended until the call results are returned (the thread enters a non-executable state, in which case the CPU does not allocate a time slice to the thread, that is, the thread pauses). Functions are returned only after the result is obtained.

2. For non-blocking will not hang, directly execute the next program, return the results and then return to process the returned value.

Python 3.4 began to support asynchronous IO programming, provided the Asyncio library, but 3.4 in the yield from @asyncio.coroutine和 this and the original generator keyword yield bad distinction, in 3.5, the use of async (for Ctrip) and await keyword, so it's better to differentiate.

The purpose of this blog is to read a bunch of information on the Internet, I still do not understand how to do Asyncio programming, use their own code, or there are various errors, so intend to look at the official manual of Asyncio and good comb, hope time and ~

Https://docs.python.org/3/library/asyncio.html

This side uses a simple official example to illustrate the order in which async and await are executed.

Import Asyncioasync def compute (x, y):    print ("Compute%s +%s ..."% (x, y))    await Asyncio.sleep (1.0)    return x + Yasync def print_sum (x, y):    result = await compute (x, y)    print ("%s +%s =%s"% (x, y, result)) loop = ASYNCIO.G Et_event_loop () Loop.run_until_complete (Print_sum (1, 2)) Loop.close ()

If async and await are not used, the order of the program is well understood

1.print_sum passes the parameter to the function print_sum

2. First sentence is executed first, result = Compute (x, y),

3. Pass 1, 2 to the compute function, and the COMPUTE function receives the parameter

4. First print ("Compute%s +%s ..."% (x, y)) prints out Compute 1 + 2 ...

5. Execute Sleep (1.0) program suspend for one second

6. Return the value of 1+2 3

7.print_sum result receives a return value of 3, executes print ("%s +%s =%s"% (x, y, result)), prints out 1 + 2 = 3

8. End of program

What is the order of execution if it is asynchronous?

Is the description of his official document:

To understand, first of all we have to understand event_loop,feture,task and other concepts, detailed can refer to official documents or http://my.oschina.net/lionets/blog/499803

First, the Get_event_loop () method lets us get a message loop, which consists of two parts: the event and Loop

Learning and using the Python Asyncio Library