Python Asynchronous Programming Asyncio (millions of concurrent)

Python because of the existence of Gil (Global lock), can not play the advantages of multi-core, its performance has been plagued by criticism. However, in IO-intensive network programming, asynchronous processing can increase hundreds of times more efficient than synchronous processing, bridging the Python performance gap, such as the latest MicroServices framework japronto,resquests per second up to millions.

One advantage of Python is that the library (third party libraries) is extremely rich and easy to use. Asyncio is python3.4 version introduced to the standard library, python2x did not add this library, after all, python3x is the future ah, haha! python3.5 also added the async/await feature.

Before we learn Asyncio, we will first understand the concept of synchronous/asynchronous :

synchronization is the logic of completing a transaction, the first transaction is executed, and if it is blocked, it waits until the transaction completes, executes the second transaction, executes the order ...

Async is relative to synchronization, which means that after the transaction is processed, it does not wait for the result of the transaction, processing the second transaction directly, and notifies the caller to process the result by State, notification, callback.

First, Asyncio

The following example compares the differences between synchronous code and asynchronous code writing , followed by a performance gap, and we use Sleep (1) to simulate an IO operation that takes 1 seconds.

Sync Code:

Import TimedefHello (): Time.sleep (1)defrun (): forIinchRange (5): Hello ()Print('Hello world:%s'% Time.time ())#any great code starts with Hello World! if __name__=='__main__': Run ()

Output: (interval is almost 1s)

Hello world:1527595175.4728756Hello World:1527595176.473001Hello World:1527595177.473494  Hello World:1527595178.4739306Hello World:1527595179.474482

Async Code:

Import TimeImportAsyncio#Defining asynchronous FunctionsAsyncdefHello (): Asyncio.sleep (1)    Print('Hello world:%s'%time.time ())defrun (): forIinchRange (5): Loop.run_until_complete (Hello ()) loop=Asyncio.get_event_loop ()if __name__=='__main__': Run ()

Output:

Hello world:1527595104.8338501Hello World:1527595104.8338501Hello World:1527595104.8338501  Hello World:1527595104.8338501Hello World:1527595104.8338501

Async def is used to define an asynchronous function that has an asynchronous operation inside it. Each thread has an event loop, and the main thread calls Asyncio.get_event_loop () to create the event loop, and you need to throw the asynchronous task to the Run_until_complete () method of the loop, The event loop schedules the execution of the synergistic program.

Second, Aiohttp

What if a concurrent HTTP request is required, usually with requests, but requests is a synchronized library, and if you want to asynchronous, you need to introduce aiohttp. this introduces a class, from aiohttp import clientsession, first to create a Session object, and then use the session object to open the Web page . The session can perform multiple operations, such as Post, get, put, head, and so on.

Basic usage:

Async with Clientsession () as session:    async with Session.get (URL) as response:

Aiohttp examples of asynchronous implementations:

ImportAsyncio fromAiohttpImportClientsessiontasks=[]url="https://www.baidu.com/{}"AsyncdefHello (URL):async with Clientsession () as Session:async with Session.get (URL) as Response:response=await Response.read ()Print(response)if __name__=='__main__': Loop=Asyncio.get_event_loop () loop.run_until_complete (hello (URL))

The Async def keyword first defines this as an asynchronous function, and the await keyword is added to the wait operation before Response.read () waits for the request response, which is a consumption IO operation. Then use the Clientsession class to initiate an HTTP request.

Multi-Link asynchronous access

If we need to request more than one URL, what to do, synchronous approach to access multiple URLs only need to add a for loop on it. But the asynchronous implementation is not so easy, on the basis of the previous need to wrap hello () in Asyncio's future object, and then pass the future object list as a task to the event loop .

Import TimeImportAsyncio fromAiohttpImportClientsessiontasks=[]url="https://www.baidu.com/{}"AsyncdefHello (URL):async with Clientsession () as Session:async with Session.get (URL) as Response:response=await Response.read ()#Print (response)            Print('Hello world:%s'%time.time ())defrun (): forIinchRange (5): Task=asyncio.ensure_future (Hello (Url.format (i))) Tasks.append (Task)if __name__=='__main__': Loop=Asyncio.get_event_loop () run () Loop.run_until_complete (asyncio.wait (tasks))

Output:

Hello world:1527754874.8915546Hello World:1527754874.899039Hello World:1527754874.90004  Hello World:1527754874.9095392Hello World:1527754874.9190395

Collecting HTTP Responses

Well, the above describes the asynchronous implementation of accessing different links, but we just make a request, if we want to collect the response one by one into a list, and then save it locally or print it out, you can collect the response all by Asyncio.gather (*tasks). , as demonstrated by the following examples.

Import TimeImportAsyncio fromAiohttpImportClientsessiontasks=[]url="https://www.baidu.com/{}"AsyncdefHello (URL): Async with Clientsession () as Session:async with Session.get (URL) as response:#Print (response)            Print('Hello world:%s'%time.time ())returnawait Response.read ()defrun (): forIinchRange (5): Task=asyncio.ensure_future (Hello (Url.format (i))) Tasks.append (Task) result= Loop.run_until_complete (Asyncio.gather (*tasks)) Print(Result)if __name__=='__main__': Loop=Asyncio.get_event_loop () run ()

Output:

Hello world:1527765369.0785167Hello World:1527765369.0845182Hello World:1527765369.0910277  Hello World:1527765369.0920424Hello World:1527765369.097017[b'<! DOCTYPE html>\r\n<!--STATUS ok-->\r\n ...

Exception resolution

If you reach 1000 concurrent, the program will error: Valueerror:too many file descriptors in select (). The reason for this error is because the Python-tuned select has a maximum length limit for open file characters. Here we have two ways to solve This problem:1. We can limit the number of concurrent quantities . Do not plug so many tasks at a time, or limit the maximum number of concurrent. 2. We can use the callback method . This is a personal recommendation to limit the number of concurrent methods, set the number of concurrent 500 or 600, processing faster.

#Coding:utf-8ImportTime,asyncio,aiohttpurl='https://www.baidu.com/'AsyncdefHello (url,semaphore): Async with Semaphore:async with Aiohttp. Clientsession () as Session:async with Session.get (URL) as response:returnawait Response.read () asyncdefrun (): Semaphore= Asyncio. Semaphore (500)#limit concurrency toTo_get = [Hello (Url.format (), Semaphore) for_inchRange (1000)]#Total 1000 Questsawait asyncio.wait (to_get)if __name__=='__main__':#now=lambda:time.time ()loop =Asyncio.get_event_loop () Loop.run_until_complete (Run ()) Loop.close ()

Python Asynchronous Programming Asyncio (millions of concurrent)