How Python concurrently handles the use of Asyncio packages

This time for you to bring Python concurrency processing Asyncio package How to use, Python concurrent processing Asyncio package use of the attention of what, the following is the actual case, together to see.

Introductory remarks: This article records my key knowledge and personal experience in the control process of Python based learning, and intends to get started with the friends of Python to learn and communicate with each other.

This article focuses on:

1, understand the function of Asyncio package and use method;
2, understand how to avoid blocking calls;
3, learn to use the process to avoid callback hell.

One, using Asyncio package to do concurrent programming

1. Concurrency and parallelism

Concurrency: Handle multiple things at once.
Parallel: Do multiple things at once.
Concurrency is used to develop scenarios to address possible (but not necessarily) parallel problems. Concurrency is better.

2. Asyncio Overview

Learn about the 4 features of Asyncio:

1. The Asyncio package uses the event loop-driven coprocessor to implement concurrency.

2. The process for Asyncio API must use yield from in the definition body, not yield.

3. Using the Asyncio process, you need to use @asyncio.coroutine decorations on the definition body. The function of decoration is to highlight the process, and when the co-process does not produce a value, the co-process will be garbage collected.

4. Python3.4, the Asyncio package only supports TCP and UDP protocols directly. If you want to use Asyncio to implement HTTP clients and servers, you often use the Aiohttp package.

There are two points to note when using yield from in the process:

1. Multiple threads that use the yield froml link will eventually have to be driven by callers that are not collaborators, call the next () function or the. Send () method on the outermost delegate generator, either callers or implicitly.

2. The sub-generators of the most inner layers in the chain must be simple generators (yield only) or objects that can be iterated.

However, there are two details to be noted in using yield from in the API of the Asyncio package:

1. The process chain written in the Asyncio package is always driven by passing the outermost delegation generator to a function in the Asyncio package API, such as Loop.run_until_complete (). That is, the co-process is not driven by invoking the next () function or the. Send () method.

2. The co-process chain is ultimately delegated to a Asyncio package or a co-process method through the yield from. That is, the inner-most sub-generators are functions that actually perform I/O operations in the library, rather than the functions we write ourselves.

Example--animate the text rotation pointer via the Asyncio package and the co-process:

Import Asyncioimport Itertoolsimport Sys@asyncio.coroutine # The process to be handed over to the Asyncio to use @asyncio. Coroutine trim def spin (msg): For Char in Itertools.cycle (' |/-\\ '): status = char + "+ msg print (status) Try:yield from Asyncio.sleep (. 1)    # Use yield from Asyncio.sleep (. 1) instead of Time.sleep (. 1), so that hibernation does not block the event loop. Except Asyncio. Cancellederror: # If the spin function wakes up and throws Asyncio.      The cancellederror exception is caused by a cancellation request being made and therefore exiting the loop.  Break@asyncio.coroutinedef slow_function (): # The Slow_function function is a co-process that uses yield from to continue the event loop when pretending to do I/O with hibernation.  # pretending to wait for I/O for a while yield from Asyncio.sleep (3) # yield from asyncio.sleep (3) expression gives control to the main loop and resumes the process after hibernation. Return 42@asyncio.coroutinedef Supervisor (): # Supervisor function is also co-spinner = Asyncio.async (Spin (' thinking! ')) # asyncio.as  The Ync (...) function schedules the run time of the spin, wraps the spin process with a Task object, and returns immediately. Print (' Spinner object: ', spinner) result = yield from slow_function () # Drives the Slow_function () function. When finished, gets the return value.  At the same time, the event loop continues to run because the Slow_function function finally uses the yield from asyncio.sleep (3) expression to turn control back to the main loop. Spinner.cancThe El () # Task object can be canceled, and Asyncio will be thrown at yield where the coprocessor is currently paused. Cancellederror exception.  The association can catch the exception, delay the cancellation, or even refuse to cancel. return resultif name = = ' main ': loop = Asyncio.get_event_loop () # Gets the reference to the event loop result = Loop.run_until_complete (Supervisor (  ) # Drive Supervisor, let it run, the return value of this process is the return value of this call. Loop.close () print (' Answer: ', result)

3. Thread vs. co-process

Thread: The scheduler can break threads at any time. Retention locks must be remembered. To protect important parts of the program from being interrupted during execution to prevent the data from being in an invalid state.

Co-process: The default is to do a full range of protection to prevent interruptions. For the coprocessor to say that there is no need to retain locks and synchronize operations between multiple threads, the coprocessor itself synchronizes because there is only one process running at any given time.

4, from the period objects, tasks and the process of output

In the Asyncio package, the phase object and the association are closely related, because yield from Asyncio can be used. The outcome of the future object. This means that if Foo is a co-function or a normal function that returns a future or a task instance, you can write this: Res=yield from foo (). This is one of the reasons that many parts of the Asyncio package can be exchanged for co-processes and periods.

Second, avoid blocking-type calls

1, there are two ways to avoid blocking calls to abort the entire application process:

1. Each blocking operation is run in a separate thread.

2. Converts each blocking operation into a non-blocking asynchronous call.

Using multithreading to handle large numbers of connections consumes too much memory, so callbacks are typically used to implement asynchronous calls.

2. Use the Executor object to prevent blocking event loops:

Use Loop.run_in_executor to delegate blocked jobs (such as saving files) to the thread pool.

@asyncio. Coroutinedef download_one (CC, Base_url, semaphore, verbose):  try: With    (yield from semaphore):      Image = Yield from Get_flag (Base_url, cc)  except Web. Httpnotfound:    status = Httpstatus.not_found    msg = ' not found '  except Exception as exc:    raise Fetcherror (cc) from exc  else:    loop = Asyncio.get_event_loop () # Gets the reference to the event loop object    Loop.run_in_executor (None, # None uses the default Trreadpoolexecutor instance        save_flag, image, Cc.lower () + '. gif ') # Incoming callable object    status = Httpstatus.ok    msg = ' OK ' 
  if verbose and msg:    print (CC, msg)  return Result (status, CC)

Asyncio a Threadpoolexecutor object behind the event loop, we can call the Run_in_executor method and send the callable object to it for execution.

Iii. from the callback expiration and the co-process

Callback Hell: If an operation needs to rely on the result of a previous operation, it has to nest the callback.

Callback Hell in Python:

def stage1 (response1):  request2 = Step1 (response1)  api_call2 (Request2, Stage2) def stage2 (RESPONSE2):  Request3 = Step2 (response2)  api_call3 (Request3, Stage3) def stage3 (RESPONSE3):  step3 (RESPONSE3) api_call1 ( Request1, Step1)

Asynchronous programming using the co-and yield from structures without callbacks:

@asyncio. Coroutinedef three_stages (request1):  response1 = yield from api_call1 ()  request2 = Step1 (RESPONSE1)  Response2 = yield from api_call2 (request2)  request3 = Step2 (response2)  response3 = yield from Api_call3 ( REQUEST3)  step3 (RESPONSE3) loop.create_task (Three_stages (REQUEST1)) # cannot be called directly, the execution time of the specified coprocessor must be displayed with an event loop. Or use the yield from expression to activate it in other processes that schedule execution time

Iv. using the Asyncio package to write the server

1. TCP and HTTP servers can be implemented using the Asyncio package

2. Web services will become an important usage scenario for Asyncio packages.

Believe that you have read the case of this article you have mastered the method, more exciting please pay attention to the PHP Chinese network other related articles!

Recommended reading:

How Python strings are converted to two-dimensional arrays

How to export Excel table functions with Vue