The async process is too hung! Take a pro test! Simply Perfect, Python Asynchronous association of Sunflower Treasure!

2.1 Blocking

Incoming group: 125240963 to get dozens of sets of PDFs Oh!

A blocking state is a state that is suspended when a program does not get the compute resources it needs. While the program is waiting for an operation to complete, it cannot continue to do anything else, it says the program is blocked on that operation.

Common blocking forms are: network I/O blocking, disk I/O blocking, user input blocking, and so on. Blocking is ubiquitous, including the CPU switching context, when all processes do not really do things and they are blocked. If it is a multi-core CPU, the kernel that is performing context switching operations is not exploited.

2.5 Multi-process

Multi-process is to take advantage of CPU multi-core advantage, at the same time to perform multiple tasks in parallel, can greatly improve execution efficiency.

2.6 Co-process

Co-process, called Coroutine, also known as micro-threading, fiber, the process is a user-state lightweight thread.

3.1 Defining the co-process

First, let's define a process, and experience the difference between its implementation and the normal one, with the following code:

First we introduce the Asyncio package so that we can use async and await, and then we use async to define an Execute () method that takes a numeric argument and prints the number after the method executes.

Here we define the Loop object, then call its Create_task () method to convert the Coroutine object to a Task object, and then we print it out and find it to be the pending state. We then add the Task object to the event loop and then we print out the Task object, and we see that its state becomes finished, and that its result becomes 1, that is, the return result of the Execute () method that we have defined.

Found the effect is the same.

3.2 Binding Callbacks

Alternatively, we can bind a callback method to a task to see the following example:

Here we define a request () method that asks for Baidu and returns a status code, but we don't have any print () statements in this method. We then define a callback () method that takes a parameter, is a task object, and then calls the print () method to print the result of the Task object. So we've defined a Coroutine object and a callback method, and we now want the effect to execute the declared callback () method when the Coroutine object finishes executing.

3.3 Multi-task co-process

In the example above, we only perform one request, what if we want to execute multiple requests? We can define a task list and then execute it using the Asyncio Wait () method, as shown in the following example:

You can see that five tasks were executed sequentially and the results were run.

3.4 Implementation of the co-process

Said in front of such a pass, is also async, is Coroutine, is a task, but also callback, but does not seem to see the advantages of the association? Instead of writing more strange and troublesome, do not worry, the above case is only for the use of the back to pave the way, next we formally see the process in the solution IO-intensive tasks have what advantages!

Here we define a Flask service, the main entry is the index () method, the method calls the sleep () method to sleep for 3 seconds, and then return the result, that is, each request this interface at least 3 seconds, so that we simulated a slow service interface.

Can be found and the normal request is not the same, still executed sequentially, time consuming 15 seconds, the average one request takes 3 seconds, good asynchronous processing it?

In fact, to achieve asynchronous processing, we have to have a pending operation, when a task needs to wait for the IO results, you can suspend the current task, instead of performing other tasks, so that we can make full use of the good resources, the above method is a well-threaded walk down, not even a hang, how can realize asynchronous? Think too much.

The Response returned by Reqeusts does not conform to any of the above conditions, so the above error is reported.

Then some of the small partners found, since the await can be followed with a Coroutine object, then I use async to change the method of the request to Coroutine object is not OK? So rewrite it to look like this:

Still, it's not asynchronous, which means it's not possible to encapsulate code that involves IO operations into async-modified methods! We have to use a request mode that supports asynchronous operations to achieve true asynchrony, so here's where Aiohttp comes in handy.

3.5 using Aiohttp

Aiohttp is a library that supports asynchronous requests, and with Asyncio we can easily implement asynchronous request operations.

The installation method is as follows:

It worked! We found that the request took 15 seconds to 3 seconds, and the time elapsed became the original 1/5.

In the code we used the await, followed by the Get () method, in the execution of these five processes, if an await is encountered, then the current process will be suspended, to perform other processes, until the other process is suspended or executed, and then the next implementation of the process.

The last run time is also about 3 seconds, of course, the extra time is IO delay.

It can be seen that, with the use of asynchronous co-processes, we are able to achieve hundreds of times of network requests in the same time, using this in the crawler, the speed increase is very considerable.

3.6 vs. single-process, multi-process

Perhaps some of the small partners are very interested to know the above example, if 100 requests, not with the asynchronous association, the use of single process and multi-process will take a lot of time, we have to test:

First, test the time of the single process:

It can be seen that multiprocessing is more efficient than a single thread.

3.7 Integration with multiple processes

Since asynchronous and multi-process have promoted network requests, why not combine them? On the latest Pycon 2018, John Reese from Facebook introduced the features of Asyncio and multiprocessing, and developed a new library called Aiomultiprocess, which is interesting to understand: http s://www.youtube.com/watch?v=0kxalh8fz3k.

This library is installed in the following ways:

The async process is too hung! Take a pro test! Simply Perfect, Python Asynchronous association of Sunflower Treasure!