Python Concurrency Practice _02_ co-process

There are many ways to implement concurrency in Python, multi-process concurrency can really take advantage of multi-core resources, and multithreading concurrency to achieve the sharing of in-process resources, but in Python due to the existence of Gil, multithreading is no way to truly achieve multi-core resources.

For compute-intensive programs, you should use multi-process concurrency to take full advantage of multicore resources, and in IO-intensive programs, the multi-core advantage is not obvious, even because most of the time is in the IO jam state, the switching consumption of multi-process is more inefficient program.

When you need to process IO-intensive tasks concurrently, you need to use the coroutine. There is no system-level scheduling, but the user-level scheduling, to avoid the overhead of system calls, although the process is ultimately serial work, but can achieve very large concurrency.

Reference article:

https://blog.tonyseek.com/post/event-manage-with-greenlet/

Producer-consumer

With the yield generator, you can simply demonstrate how the process works:

Import time
DEF consumer ():
Print "Ready to receive"
While True:
y = (yield)
Time.sleep (1)
Print "Receive%s from producer"%y
Def producer ():
c = Consumer ()
C.next ()
i = 1
While I > 0 and I < 11:
Time.sleep (1)
Print "Send%s to consumer"%i
C.send (i)
i + = 1
if __name__ = = ' __main__ ':
Producer ()

The above process shows the basic producer-consumer model, the consumer consumer is a generator;

When C.next () is called in producer for the first time, the consumer is activated and the co-process (consumer) is suspended when it is run to yield, waiting for the generator to be called next or send.

Producer carries out subsequent operations and enters a loop, after each pause of 1s, send a message to the generator, the consumer yield gets to the message, and the subsequent work is done.

As you can see, each yield needs to wait for the send incoming message before continuing with the task after it is executed.

Through yield to achieve the co-process

Now it's time to actually create a co-process with yield.

You can imagine a model where there are a lot of similar tasks on a site, and there will be a steady flow of these tasks, with a foreman (foreman) in charge of the site, the foreman in order to assign a task to the worker (worker), Will develop a set of processes (pipeline) to facilitate management: the allocation of workers, acceptance work (accept), because the worker's work time is far greater than the time allotted tasks, the work of these workers (simple and tedious repetitive labor) as IO operations, this is an IO-intensive task. Let's look at how Python works through yield to achieve the complete work of the process:

1 defMain ():2 Foreman (Args_of_overall,worker_num)3 4 defForeman (Args_of_overall,worker_num):5Pipeline =create_pipeline (args_of_pipeline,worker_num)6      forI,jobinchEnumerate (Get_jobs (args_of_ceate_jobs)):7worker_id = i%Worker_num8 Pipeline.send ((job,worker_id))9 Ten @coroutine One defworker (pipeline,accepting,job,my_id): A      whileTrue: -Args_of_job, worker_id = (yield ) -         ifworker_id = =my_id: theresult =Work (Args_of_job) - accepting.send (Result) -         elifPipeline is  notNone: - Pipeline.send ((job,worker_id)) +  - @coroutine + defAccept (): A      whileTrue: atresult = (yield ) -         #do_some_accepting -  - defCreate_pipeline (args_of_pipeline,worker_num): -Pipeline =None -accepting =Accept () in      forwork_idinchRange (work_num): -Pipeline =worker (pipeline,accepting,job,work_id) to     returnPipeline +  - defget_jobs (args_of_ceate_jobs): the      forJobinchJob_source: *         yieldJob $ Panax Notoginseng defCoroutine (func): -     defWarper (*args): theF = Func (*args) + F.next () A         returnF the     returnWarper +  - defWork (args_of_job): $     Pass $     #do_some_work -  - if __name__=='__main__': theMain ()

In the above process, the worker and the acceptance work are the co-processes, and the get_jobs () function is a generator, when the job is dynamically added, it can be rewritten as a co-process, thus, the generator should be a special association.

All of the above work is done serially, although there are many workers, the work between workers is concurrent (IO wait time), but the work has always been from the first to start one assignment task.

Python Concurrency Practice _02_ co-process