Python learning asynchronous Io[all for Python---]

1.1.1. Pre-environmental preparedness and fundamentals

Installation:

PIP3 Install Aiohttp

PIP3 Install Grequests

PIP3 Install Wheel

PIP3 Install Scrapy

Attention:

Scrapy on Windows relies on https://sourceforge.net/projects/pywin32/files/

Installing twisted

A. http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted,

B. Download: TWISTED-17.1.0-CP35-CP35M-WIN_AMD64.WHL

C. Enter the directory where the file is located

D. PIP3 Install TWISTED-17.1.0-CP35-CP35M-WIN_AMD64.WHL

How IO operations are implemented

Why do we need asynchronous requests?

In the case of a normal request, a request ends before the next request is opened [serial request], and if there is one request at a time, subsequent requests are terminated.

If it is a multi-threaded asynchronous request, multiple requests are opened by multiple threads at the same time, and one requested exception does not affect the other

There are 3 ways to implement IO operations:

synchronizing "serial operations"
Multi-process "more resource-intensive, with operating system calls"-more suitable for computationally dense operations because of the need for concurrent operations, consuming CPU
Threads are the smallest unit of computer work
multithreading "has CPU calls, saves resources" and is more suitable for multi-IO operations because CPU resources are not consumed after sending requests
There is at least one thread in the process, and the default has a primary thread and the internal resources of the shared process
more than one process, a single thread completes multiple tasks "can receive multiple requests at the same time and then process requests in one single"
If a block is encountered, the next request is executed, and if the blocked request receives a reply, the request "callback implementation", which was just blocked, is much more efficient than multithreading.

Note: There is a Gil "Global interpreter Lock" inside the thread, and Python has a Gil lock [which guarantees that only 1 threads are allowed to operate in 1 processes simultaneously] and does not allow the CPU to operate multiple threads. No CPU calls are allowed [that is, the CPU is limited, multithreading is limited]. But threads can do IO operations, multiple threads can do multiple IO operations at the same time [URL requests, etc., because the CPU only needs to be sent, not consume CPU resources after sending],

Asynchronous operation using multithreading to implement IO:

Import requestsfrom concurrent.futures.thread Import Threadpoolexecutorpool = Threadpoolexecutor (5) # Create thread pool, Can also be understood as multithreading here url_list = [    ' https://www.baidu.com/',    ' https://www.taobao.com/',    ' https:// Www.google.com/search ',    ' https://hao.360.cn/',]def async_url (URL):    try:        response = requests.get (URL)        print (' Normal request: ', ' ', ' url, ' ' ', response.content ')    except Exception as E:        print (' Exception request: ', E ' for the URL in url_list:    print (' request start: ', url)    pool.submit (async_url, URL) pool.shutdown ()   # Close Thread

Background Display results:

Asynchronous operation with multiple processes for IO:

[Other Ibid]from concurrent.futures.process import processpoolexecutorimport requestspool = Processpoolexecutor (5) # Create a process pool, Can also be understood as multithreaded here Pool.submit (Async_url, URL)   # Async_url is a method, URL is passed past parameters Pool.shutdown ()   # Close Process

Asynchronous Io_1---asyncio module (no-http)

Python Learning async for---io [asyncio module (no-http)]

Asynchronous Io_2---gevent+grequests

Python learns asynchronous---io [gevent+grequests module]

Asynchronous Io_3---Twisted module

Python learns asynchronous---io [twisted module]

Asynchronous Io_4---tornado module

Python learns asynchronous---IO [tornado module]

Custom Asynchronous IOPython Learning---io async [custom Asynchronous IO]

Python learning asynchronous Io[all for Python---]