Python-based multi-process shared variables are opened correctly

Below for you to share a Python-based multi-process shared variable right open way, with a good reference value, I hope to be helpful to everyone. Come and see it together.

Multi-process sharing variables and getting results

Because of engineering needs, to use multi-threading to run a program. But because I heard that Python's multithreading is fake, so use multi-process, anyway the task needs to share less parameters.

Access to data, found that the implementation of multiple processes mainly using multiprocessing, there are two ways, one is process, the other is pool.

p = Process (target=fun,args= (args))

The P.start () is used to start a child process, and the P.join () method is used to make the child process run and then the parent process is executed.

But it's annoying to write a for loop to open n threads and joins.

So I recommend pool. It can open a fixed-size process pool, then each thread executes the function called by the Apply_async () function, and finally close and join.

The code is as follows:

Pathm=manager (). Queue (Len (pathlist)) for D in Pathlist:pathm.put (d) p=pool (cp.threads) results=[]for I in Range (cp.threads): temp= P.apply_async (processworker,args= (I,PATHM,CP)) results.append (temp) print ' Waiting for all subprocesses done ... ' P.close () p.join () print ' All subprocesses finish processing. ' Results=[r.get () for R in results]

The code above shows how to use the pool multi-process, how to share variable pathm between processes in the pool, and how to get the results of the process function execution. It is important to note that the Processworker must be an unbounded function, otherwise the function cannot be pickle and cannot be assigned to each process.

CPickle.PicklingError:Can ' t pickle <type ' instancemethod ';: Attribute lookup __builtin__.instancemethod failed

Bounded functions and Python's multi-process mechanism

From the above to a concept, is bounded function of the concept of unbounded functions.

After reviewing the information, I summarize as follows:

Bounded functions are wrapped in a class, and only functions that are used after the class is instantiated are the bounds of that instance. We often refer to these functions as class methods. For example, a class method with the self argument.

An unbounded function can be a function that is not wrapped in a class, or a static method in a class that is independent of the class. As a static method in a class, it is defined in a class, but cannot access parameters and other methods in the class.

The Python multi-process mechanism should be to compile and package the methods to be called by each process and the parameters passed in (as in the example above), and then copy them into each process to execute processworker. If the input is a bounded function, then its parameters should be the class (including parameters and methods) it belongs to, but this is not available, and the class properties and methods can have pits, resulting in difficult to package. So python limits the number of processes to be called by a function that cannot be a class method.

We're going to put the multi-process call function out of the class or into a static function. But a static function cannot be called by the method of the owning class (self. Processworker form) that needs to be called externally, such as Mc=myclass (), MC. Processworker to invoke, or MyClass (). Processworker to invoke.