"Python" co-process

Co-process, also known as micro-threading, fiber. English name Coroutine.

The concept of the co-process was raised early, but it has not been widely used in some languages (such as LUA) in recent years.

subroutines, or functions, are hierarchical calls in all languages, such as a call to b,b in the execution of a call to C,c execution complete return, b execution is finished, and finally a executes.

So the subroutine call is implemented through the stack, and a thread is executing a subroutine.

Subroutine calls are always one entry, one return, and the call order is clear. And the call of the association and subroutine is different.

The process appears to be a subroutine, but during execution, it can be interrupted inside the subroutine, then execute another subroutine, and then return to execute at the appropriate time.

Note that interrupts in one subroutine, to execute other subroutines, are not function calls, a bit like CPU interrupts. such as subroutines A, B:

defA ():Print('1')    Print('2')    Print('3')defB ():Print('x')    Print('y')    Print('Z')

Assuming that the process is executed by the coprocessor, it can be interrupted at any time during the execution of a, and the execution of b,b may be interrupted during execution to execute a, and the result may be:

xy3z

But in a there is no call to B, so the call of the association is more difficult to understand than the function call.

It seems that a, B's execution is a bit like multi-threading, but the process is characterized by a thread execution, and multithreading ratio, what is the advantage of the co-process?

The greatest advantage is the high execution efficiency of the process. Because the subroutine switch is not a thread switch, but is controlled by the program itself, therefore, there is no thread switching overhead, and multithreading ratio, the more the number of threads, the performance advantage of the association is more obvious.

The second big advantage is that there is no need for a multi-threaded lock mechanism, because there is only one thread, there is no simultaneous write variable conflict, in the process of controlling shared resources without locking, only need to judge the state is good, so the execution efficiency is much higher than multithreading.

Because the co-process is a thread execution, how do you take advantage of multicore CPUs? The simplest method is multi-process + co-progression, which takes full advantage of multicore, and maximizes the high-efficiency of the co-processes, which can achieve very good performance.

Python support for the process is implemented through generator.

In generator, not only can we for iterate through loops, but we can constantly call next() the function to get yield the next value returned by the statement.

But Python yield can not only return a value, it can also receive parameters emitted by the caller.

See Example:

The traditional producer-consumer model is a thread-write message, a thread that takes messages, controls the queue through a lock mechanism, and waits, but can be accidentally deadlocked.

If the use of the process, the producer produces the message, directly by yield jumping to the consumer to start execution, after the consumer has finished, switch back to the producer to continue production, the efficiency is extremely high:

defconsumer (): R="'     whiletrue:n=yieldRif  notN:return        Print('[CONSUMER] consuming%s ...'%N) R='OK'defProduce (c): C.send (None) n=0 whileN < 5: N= n + 1Print('[PRODUCER] producing%s ...'%N) R=c.send (n)Print('[PRODUCER] Consumer return:%s'%r) c.close () C=consumer () produce (c)

Execution Result:

[PRODUCER] Producing 1... [CONSUMER] Consuming1... [PRODUCER] Consumerreturn: 200Ok[producer] Producing2... [CONSUMER] Consuming2... [PRODUCER] Consumerreturn: 200Ok[producer] Producing3... [CONSUMER] Consuming3... [PRODUCER] Consumerreturn: 200Ok[producer] Producing4... [CONSUMER] Consuming4... [PRODUCER] Consumerreturn: 200Ok[producer] Producing5... [CONSUMER] Consuming5... [PRODUCER] Consumerreturn: $ OK

Notice that the consumer function is a generator , put one in consumer produce after:

1. First call the c.send(None) startup generator;

2. Then, once the thing is produced, by c.send(n) switching to consumer execution;

3. consumerBy yield getting the message, processing, and passing the yield results back;

4. produceGet consumer The result of processing, continue to produce the next message;

5. producedecided not to produce, by c.close() closing consumer , the whole process was over.

The entire process is unlocked, executed by one thread, produce and the consumer task is completed collaboratively, so called "co-process", rather than a thread-preemptive multi-tasking.

Finally, to apply Donald Knuth's sentence summarizes the characteristics of the association: "Subroutine is a special case of the process." ”

"Python" co-process