Python concurrent Programming: Introduction to the process

A primer

Based on a single thread to implement concurrency, that is, only one main thread (obviously available CPU only one) to implement concurrency, first review the nature of Concurrency: Switch + save state CPU is running a task, in two cases will be cut off to perform other tasks (switch is controlled by the operating system), One scenario is that the task is blocked, and the other is that the task is taking too long to calculate or having a higher-priority program instead

　　

PS: When introducing the process theory, the three execution states of the process are mentioned, and the thread is the execution unit, so it can also be understood as the three states of the thread

　　

One: The second situation does not improve efficiency, just let the CPU can rain equitably, to achieve the seemingly all tasks are "simultaneous" effect, if more than one task is pure calculation, this switch will reduce efficiency. For this we can verify based on yield. Yield itself is a way to save the running state of a task in a single thread, so let's review it briefly:

1, yiled can save the state, the state of yield is similar to the save thread state of the operating system, but yield is code-level control, more lightweight 2, send can pass the result of a function to another function, so as to realize the switch between the single-threaded program

　　

Simply switching will reduce operational efficiency

# Serial Execution Import timedef consumer (res):    ' Task 1: Receive data, process data '    passdef producer ():    ' Task 2: Generate Data '    res = [] For    i in range:        res.append (i)    return resstart = Time.time () # serial Execution res = producer () consumer (res)       # Write Consumer (producer ()) To reduce execution efficiency stop = Time.time () print (Stop-start)   # 1.835823059082031

　　

Concurrent execution

# based on yield concurrency execution import timedef consumer ():    ' Task 1: Receive data, process data ' while    True:        x = Yielddef producer ():    " Task 2: Production data '    g = consumer ()    next (g)    for I in range:        g.send (i) start = Time.time () # Save state based on yield Implement two tasks to switch directly back and forth, that is, the concurrent effect of # PS: If each task is added to print, then obviously see two tasks of printing is you one time I, that is, the concurrent execution of producer () Stop = Time.time () print (Stop-start) # 2.503395080566406

Two: The first case of the switch. In the case of a task encountered Io, cut to the task two to execute, so that the task can be used to block the time to complete the task two calculation, the efficiency of the tip is that this

Yield does not meet IO switching

Import Timedef Consumer ():    ' Task 1: Receive data, process data ' while    True:        x = Yielddef producer ():    ' Task 2: Production data '    g = consumer ()    next (g) for    I in range:        g.send (i)        time.sleep (2) start = Time.time () producer ()  # Concurrent execution, but task producer encounters IO will block and will not switch to other tasks within that thread to execute stop = Time.time () producer (Stop-start)

　　

　　For single-threaded, we inevitably have IO operations in the program, but if we can control multiple tasks in one single thread in our own program (that is, the user program level, not the operating system level), switch to another task to calculate when a task encounters IO blocking, This ensures that the thread is in the most ready state, that it can be executed at any time by the CPU, and that we hide the maximum amount of our IO operations at the user program level, which can confuse the operating system and let it see that the thread seems to have been calculating, the IO is relatively small, So that more of the CPU's execution rights are assigned to our thread

The nature of the process is that in a single thread, the user controls a task by itself when the IO block is switched on to another task to execute, to improve efficiency. To achieve this, we need to find a solution that can meet the following conditions

　　

1, can control the switch between multiple tasks, before switching to save the state of the task, in order to re-run, you can continue to run based on the paused position 2, as a supplement to 1, can detect IO operation, in the case of IO operation will occur before switching

　　

Introduction of two-way co-process

Co-process: is a single-threaded concurrency, also known as micro-threading, fiber

A sentence description is the association process: The process is a user-state of a lightweight thread, that is, the association process is controlled by the user program

It should be emphasized that:

1, Python threads belong to the kernel level, that is, by the operating system control scheduling (such as single-threaded encounter IO or long execution time will be forced to hand over the CPU execution rights, switch other threads to run) 2, single-line range open the process, once encountered Io, will be from the application level (not the operating system) control switch, To improve efficiency (non-IO switching is independent of efficiency)

Compared to the operating system control thread switching, the user in a single-threaded control of the switch process

The advantages are as follows:

1, the transition cost of the association is smaller, belongs to the program level of the switch, the operating system is not fully aware of, and therefore more lightweight 2, single-threaded can achieve the effect of concurrency, to maximize the use of CPU

　　

Disadvantages are as follows:

1, the nature of the process is single-threaded, unable to use multicore, can be a program to open multiple processes, each process to open multiple threads, each process to open multiple threads, each line range open 2, the association refers to a single thread, so once the association is blocked, it will block the entire thread

　　

Summary of the characteristics of the process:

1. Concurrency must be implemented in only one single thread

2. No lock required to modify shared data

3. The user program to save multiple control Liu's context stack

4. Attach: A co-process encountered IO operation automatically switch to other co-process (how to implement the detection io,yield,greenlet can not be implemented, the use of the Gevent module (select mechanism))

# based onyieldConcurrent execution
ImportTime


defConsumer():
    " "Task1: Receive data, process data""
While True:
        x=Yield


defproducer():
    " "Task2: Production Data""
g = Consumer ()
    Next(g)
     forIinchRange( -):
G.send (i)


Start = Time.time ()
# based onyieldsave state, enable two tasks to switch directly back and forth, that is, the effect of concurrency
# PS: If you add a print to each task, you can see clearly that the print of two tasks is the one I once, which executes concurrently
Producer ()


Stop = Time.time ()
Print(Stop-start)

Python concurrent Programming: Introduction to the process