A review of the process communication between Python multiprocessing modules

This period of time addicted to multiprocessing module can not extricate themselves, no way, the basis of Python is not very familiar, so is constantly encountered problems to solve the problem. Before learning Asyncio module smattering, later thought multiprocessing module is smattering, take advantage of the summer vacation bored to study a bit, have to say, this deepened their grasp and understanding of the Python foundation ... So we have this series of "Python standard library of Multiprocessing Library Research (1)" Python multiprocessing standard library use queue communication points of attention " A study of the multiprocessing Library of the Python Standard library (2), today is the final summary of it, as well as the doubts, then you will know the solution of the confusion.

I did not explain in detail multiprocessing the source code of this package, did not explain the pool underlying mechanism, there is only one reason, I am not enough heat, if now forced to understand, I do not have much benefit, so I choose to shun its nature, first the most basic usage master, really do not know how to look at the source code, Probably over the source code, its implementation mechanism has a general understanding. Multiprocessing Source code Package I probably took a serious look at the two pool.py and managers.py,multiprocessing queues.py probably looked at, the final summary is: Do not feel those what package, frame, module tall, more convenient, in fact, read the source carefully You will find that it is actually a solid foundation or a group of well-grounded Daniel wrote it. In other words, those tall frames are made up of things that are basic. Some people say, you say so easy, you write a try. I'm sorry, I can't write, I said these are not what I pretend to do, just to explain the facts, and do not belittle the meaning of those frameworks, just want to reduce your "awe" of the framework of the heart, but then again, I think I can write on the framework, but now is not the time, the foundation is not solid. The writing of a framework requires the depth and breadth of the author's knowledge! Unfortunately, I am not qualified enough, slowly come, I believe that I will have a day ...

Summary of the almost, specific code you can look at the above mentioned articles

Here's a few questions: Why would a process created with multiprocessing process be able to use multiprocessing's queue for interprocess communication, while the process created with pool process pools is not working? And why use the multiprocessing queue instead of the standard library's queue.

As I said in an earlier article, the standard library queue can only implement inter-thread communication, its Get,put method is blocked, and in addition to these two methods are not secure, can not guarantee that in multi-threaded communication, can be stable access to variables, the variable is processed ...

Queue.queue is an in-process non-blocking queue, and Multiprocess.queue is a cross-process communication queue. The former is private, and the latter is common to each of the sub-processes.
From queue import Queue This is a normal queue pattern, similar to the normal list, FIFO mode, the Get method blocks the request until there is data get out
From multiprocessing. Queue Import Queue This is a multi-process concurrent queue that is used to resolve inter-process communication problems. The normal queue cannot be implemented. For example, to run multi-process operations on a number of IP lists, the results of the operation will be stored in the queue, this must use the queue provided by multiprocessing to achieve

Threads are part of a process, threads run in process space, threads generated by the same process share the same memory space, and threads generated by the process are forced to exit and clear when the process exits. Threads can share all of the resources owned by a process with other threads that belong to the same process, but they do not inherently have system resources, and of course, have the resources they are required to run ... The process can not share resources without sharing ...

So, the explanation is almost right ...

Here's a look at the manager's introduction to the official website:
Managers provide a-to-create data which can be gkfx between different processes, including sharing over a network is Tween processes running on different machines. A Manager object controls a server process which manages shared objects. Other processes can access the shared objects by using proxies.

I looked at the manager's source code, which used the proxy and the Recv () method in the socket (as if it were inside the socket) for cross-process communication and proxy communication.

Here to clear a mind:
Multiprocessing.process creates a process that has a common parent process, while the process created by Multiprocess.pool is not. The process pool exists to reduce the cost of creating and destroying processes. Here I have a question: what is the difference between creating a child process and creating a process? The cost should be different.
Python implements multiple interprocess communication in many ways, such as queues, pipelines, and so on.
However, these methods only apply when multiple processes originate from the same parent process
If multiple processes do not originate from the same parent process, they can only use shared memory, semaphores, etc., but these methods are cumbersome and inflexible for complex data structures such as queue,dict,list.
Manager is a more advanced multi-process communication method that supports any data structure supported by python ....

Therefore, we are able to use multiprocessing's queue communication, but if we are using processes created by the process pool, we have to use the data structure encapsulated by the manager class ...

A review of the process communication between Python multiprocessing modules