13. multi-process multiprocessing, process pool, and multiprocessing

13. multi-process multiprocessing, process pool, and multiprocessing
Content:

Multiprocessing:

• Process Creation and running
• Common process functions

Process pool:

• Why is there a process pool?
• Process pool creation and running: Serial and parallel
• Callback Function

Multiprocessing:

The multiprocessing module is required for multiple processes in python.

 

• Multi-Process Creation and running:

1. process Creation: Process object = multiprocessing. process (target = function name, args = (parameter,) [Supplement: Because args is a tuples, add "," to a single parameter.]

2. process running: Process object. start ()

Process join is the same as thread join. It means"Blocking the current process until the process that calls the join method is executed, and then continue to execute the current process"

Note: This must be used in windows Code and is not required in Linux.

Import multiprocessing, time, osdef thread_run (): print (threading. current_thread () def run (name): time. sleep (1) print ("hello", name, "run in", OS. getpid (), "ppid:", OS. getppid () if _ name __= = '_ main _': # obj = [] for I in range (10): p = multiprocessing. process (target = run, args = ('bob',) obj. append (p) p. start () start_time = time. time () for I in obj: I. join () print ("run in main") print ("spend time:", time. time ()-start_time)

 

• Same as multithreading: You can also create a Process by inheriting the Process of multiprocessing.

The class that inherits the Process class of multiprocessing mainly does two things:

1. If you initialize your own variables, you must first call the _ init _ () of the parent class. [if you do not call these variables, you must enter relevant parameters by yourself. Please !] Then perform Initialization on your own. If you do not need to initialize your own variables, you do not need to overwrite _ init __and directly use the _ init _ of the parent class to [inherit]

2. Rewrite the run function.

import multiprocessingclass myProcess(multiprocessing.Process):    def run(self):        print("run in myProcess")if __name__=="__main__":    p=myProcess()    p.start()    p.join()

Common process functions:

• OS. getpid (): gets the current process number.
• OS. getppid (): gets the parent process ID of the current process.
• Process object. is_alive (): determines whether the process is alive.

• Process object. terminate (): terminate the process. [This method is not recommended. It is rarely used]

 

Process pool:

• Why process pool?
  • If you want to start a large number of sub-processes, you can create sub-processes in batches using the process pool, and the process pool can limit the number of processes running [too many people want to swim, the pool capacity determines the number of persons swimming.
  • The Pool class can provide a specified number of processes for users to call. When a new request is submitted to the Pool, if the Pool is not full, a new process is created to execute the request. If the process pool is full, a new process will be created to execute the requests until the process in the pool ends.

 

• Process pool creation and use:
  • To use a process Pool, you need to import: from multiprocessing import Pool
  • Create process Pool: Process Pool object = Pool (capacity)
  • Add a process to the process pool:
    • Serial: Process pool object. apply (func = function name, args = (parameter ,))

From multiprocessing import Poolimport time, osdef func1 (I): time. sleep (1) print ("run in process:", OS. getpid () if _ name __= = "_ main _": pool = Pool (5) start_time = time. time () for I in range (10): pool. apply (func = func1, args = (I,) # Serial. Add a pool after running. close () # close before waiting for the pool. join () print ("main run done, spend_time:", time. time ()-start_time)

• Parallel: Process pool object. apply_async (func = function name, args = (parameter,), callback = callback function)

From multiprocessing import Poolimport time, osdef func1 (I): time. sleep (1) print ("run in process:", OS. getpid () if _ name __= = "_ main _": pool = Pool (5) start_time = time. time () for I in range (10): pool. apply_async (func = func1, args = (I,) # parallel pool. close () # close before waiting for the pool. join () print ("main run done, spend_time:", time. time ()-start_time) #2.6, proving to be parallel

• Callback Function usage: In parallel, callback = callback function is supported. After a process is executed, the callback function is called and the parameter is the return value in func.
• Note: The callback function is executed in the parent process! [When the son finishes executing the task, he will call the function in his father]

From multiprocessing import Poolimport time, osdef func1 (I): time. sleep (1) print ("run in process:", OS. getpid () return "filename" def log (arg): # the return value of the func function in process creation is print ("log done:", arg) if _ name __= = "_ main _": pool = Pool (5) start_time = time. time () for I in range (10): pool. apply_async (func = func1, args = (I,), callback = log,) # The log parameter is the return value pool of func1. close () # close before waiting for the pool. join () print ("main run done, spend_time:", time. time ()-start_time)

• Note:PoolObject calljoin()The method is called after all sub-processes are executed.join()You must callclose(), Callclose()You cannot add newProcess. [That is, for example, if the swimming pool only sells tickets for one hour and closes the door at, you will not be able to sell tickets after and will wait until the people in the swimming pool come out and close the door, the process pool close means closing, not ending. It just closes the incoming door, the processes in the pool can also run] [The join Operation in the process pool is to wait until all the processes in the pool are executed. If the processes come in later, the process will never end. Therefore, you need to close the process first, wait until the process ends]
  • • Small test: