Python multi-process programming example analysis

This article mainly introduces the Python multi-process programming technology, including the concepts of threads, queues, synchronization, and other related skills, if you need it, you can refer to this article to analyze the Python multi-process programming technology in the form of an example, which will help you further develop the Python programming skills. Share it with you for your reference. The specific analysis is as follows:

Generally, due to restrictions on Python threads, for example, multithreading cannot fully utilize multi-core CPU, we prefer to use multi-process in Python. However, we also use multithreading in non-blocking asynchronous UI and other scenarios. This article focuses on the problem of Python multi-process.

Python introduces a multi-process mechanism in 2.6 and provides a wide range of components and APIs to facilitate the compilation of concurrent applications. The Process, Queue, Pipe, and Lock components of the multiprocessing package provide functions similar to multithreading. With these components, you can easily write multi-process concurrent programs.

Process

Process is a bit like java. lang. Thread, but Thread is a Thread. Start method is used to start a process. A simple example:

from multiprocessing import Processimport osimport timedef sleeper(name, seconds):  print "Process ID# %s" % (os.getpid())  print "Parent Process ID# %s" % (os.getppid())  print "%s will sleep for %s seconds" % (name, seconds)  time.sleep(seconds)if __name__ == "__main__":  child_proc = Process(target=sleeper, args=('bob', 5))  child_proc.start()  print "in parent process after child process start"  print "parent process abount to join child process"  child_proc.join()  print "in parent process after child process join"  print "the parent's parent process: %s" % (os.getppid())

Target and args must be specified to instantiate a Process. Target is the entry method of the new process and can be considered as the main method. Args is the parameter list of this method. Starting a process is similar to starting a Thread. you must call the start method. You can also inherit the Process, overwrite the run method, and implement the logic of the Process in the run method. Calling the join method will block the current calling process until the call process stops running.
You can call the terminate method to manually terminate a process. in UNIX systems, this method sends a SIGTERM semaphore, while in windows systems, the TerminateProcess method is used. Note that the exit processing logic is not executed, and the sub-processes of the process are not terminated. they will only become orphan processes.

Queue

Queue is a multi-process secure Queue. you can use Queue to transmit data between multiple processes. The put method is used to insert data into the queue. The put method has two optional parameters: blocked and timeout. If blocked is True (default value) and timeout is a positive value, this method blocks the time specified by timeout until the queue has space left. If it times out, a Queue. Full exception is thrown. If blocked is False but the Queue is Full, the Queue. Full exception is thrown immediately.

The get method can read from the queue and delete an element. Similarly, the get method has two optional parameters: blocked and timeout. If blocked is True (default) and timeout is a positive value, if no element is obtained during the waiting time, a Queue. Empty exception is thrown. If blocked is False, two conditions exist. if a value in the Queue is available, this value is returned immediately. Otherwise, if the Queue is Empty, a Queue. Empty exception is thrown immediately. A sample code of Queue:

from multiprocessing import Process, Queuedef offer(queue):  queue.put("Hello World")def test(queue, num):  queue.put("Hello World: " + str(num))if __name__ == '__main__':  q = Queue()  p1 = Process(target=test, args=(q, 1))  p1.start()  p = Process(target=offer, args=(q,))  p.start()  p2 = Process(target=test, args=(q, 2))  p2.start()  p2 = Process(target=test, args=(q, 3))  p2.start()  print q.get()  print q.get()  print q.get()  print q.get()  print q.close()

Output:

Hello World: 1
Hello World
Hello World: 2
None

Pipes

The Pipe method returns (conn1, conn2) representing two ends of a pipeline. The Pipe method has the duplex parameter. if the duplex parameter is True (default), the Pipe is in full duplex mode, that is, both conn1 and conn2 can be sent and received. If duplex is False, conn1 is only responsible for receiving messages, and conn2 is only responsible for sending messages.

The send and recv methods are respectively the methods for sending and receiving messages. For example, in full duplex mode, you can call conn1.send to send messages and conn1.recv to receive messages. If no message is received, the recv method will be blocked. If the pipeline has been closed, the recv method will throw an EOFError.

from multiprocessing import Process, Pipedef send(conn):  conn.send("Hello World")  conn.close()if __name__ == '__main__':  parent_conn, child_conn = Pipe()  p = Process(target=send, args=(child_conn,))  p.start()  print parent_conn.recv()

Synchronization

The multiprocessing package provides components such as Condition, Event, Lock, RLock, and Semaphore for synchronization. The following is an example of using Lock:

from multiprocessing import Process, Lockdef l(lock, num):  lock.acquire()   print "Hello Num: %s" % (num)  lock.release()if __name__ == '__main__':  lock = Lock()for num in range(20):  Process(target=l, args=(lock, num)).start()

Summary

The above is a brief introduction and example of the Python multiprocessing Library. if you are familiar with Java multi-thread development, do you think you are familiar with java Concurrency APIs? however, javaConcurrency only processes multiple threads, we can use these APIs directly based on the previous Java Multithreading experience.

If you are interested, you can test and run the examples in this article to deepen your understanding. I believe this article has some reference value for everyone's learning about Python programming.