Implementing multi-core parallel computing with Python

Normally written programs, whether single-threaded or multi-threaded, are mostly only one process, and can only work in one core. So many applications are running at full load, CPU usage in Task Manager is only 50% (dual-core CPU) or 25% (quad-core CPU)

If you can get a program to build multiple processes on its own, and let them run in parallel, you can run concurrently on different CPU cores to achieve parallel computing.

This is what Python's parallel computing does.

Here's a look at the first sample:

1 #main.py2 ImportMultiprocessing3 Import Time4 ImportNumPy as NP5  fromFuncImportWriteln6  fromCalcImportCalc7 ImportScipy.io as Sio8 9 deffunc1 (x):Ten Calc () Onec1=0 AD1=np.zeros (233, int) -      forIinchXrange (5): -D1[c1]=writeln (1, i) theC1+=1 -         #time.sleep (1) -Sio.savemat ('11.mat',{'DD':d 1}) -  + defFunc2 (x): - Calc () +C2=0 AD2=np.zeros (233, int) at      forIinchXrange (5): -D2[c2]=writeln (2, i) -C2+=1 -         #time.sleep (1) -Sio.savemat ('22.mat',{'DD':d 2}) -  in deffunc3 (x): - Calc () toc3=0 +D3=np.zeros (233, int) -      forIinchXrange (5): theD3[c3]=writeln (3, i) *C3+=1 $         #time.sleep (1)Panax NotoginsengSio.savemat ('33.mat',{'DD':d 3}) -  the defFunc4 (x): + Calc () Ac4=0 theD4=np.zeros (233, int) +      forIinchXrange (5): -D4[c4]=writeln (4, i) $C4+=1 $         #time.sleep (1) -Sio.savemat ('44.mat',{'DD':d 4}) -  the if __name__=="__main__": -Pool = multiprocessing. Pool (processes=4)        Wuyi  thePool.apply_async (Func1, (1, )) -Pool.apply_async (Func2, (2, )) WuPool.apply_async (Func3, (3, )) -Pool.apply_async (Func4, (4, )) About  $ pool.close () - Pool.join () -  -  A     Print "sub-process (es) done."

 1  #   2  def   Writeln (x, y):  3  aa=x*10+y  4   (AA)  5  return  (aa) 

1 # calc.py 2 def Calc (): 3     x=2334for in      xrange (1000000000):5         x=x+16          x=x-1

main.py

Line 49 creates a new process pool and specifies that the number of native CPU cores is 4

This allows the main program to run with 4 additional processes, each running on a different core, enabling multi-core parallelism

Line 51--54 adds the Func1--func4 four functions to the process pool.

Note that if we join more than 4 func, then only four will be running at the same time. The rest will be waiting in line.

calc.py

This is a dead loop .... is to demonstrate CPU usage ...

Operating effect:

Single calc () runtime, CPU consumption is 25%

After multiprocessor is enabled, 5 Python.exe processes (one Master + 4 sub-processes) are turned on, and the CPU consumes 100%. At the same time the fan began to turn wildly ...

Because the parallel runtime has sequential uncertainties (refer to multi-threading on the OS textbook), print output can be messy. Here we are all saved in the mat file.

With this approach, I can get my training data set running in parallel.

Reference:

http://www.coder4.com/archives/3352

Http://www.cnblogs.com/kaituorensheng/p/4445418.html

http://rfyiamcool.blog.51cto.com/1030776/1357112

Implementing multi-core parallel computing with Python