Python Processes and Threads

2017-07-30 19:44:25

What do you mean "multitasking"? To put it simply, the operating system can run multiple tasks at the same time. For example, while you're surfing the Internet with your browser and listening to MP3, while you're working with Word, that's multitasking, at least 3 tasks running at the same time. There are a lot of tasks quietly running in the background at the same time, but the desktop is not displayed.

Multicore CPUs are now very popular, but even the single-core CPUs of the past can do multitasking. Since the CPU execution code is executed sequentially, how does a single-core CPU perform multi-tasking?

The answer is that the operating system turns each task to perform alternately, Task 1 executes 0.01 seconds, switches to Task 2, Task 2 executes 0.01 seconds, then switches to Task 3, executes 0.01 seconds ... This is done repeatedly. On the surface, each task is executed alternately, but because the CPU is executing too fast, we feel as if all the tasks are executing at the same time.

True parallel multitasking can only be done on multicore CPUs, but because the number of tasks is much larger than the number of cores in the CPU, the operating system automatically shifts many tasks to each core.

For the operating system, a task is a process, such as open a browser is to start a browser process, open a notepad started a Notepad process, open two Notepad started the two Notepad process, open a word started a word process.

Some processes do more than one thing at the same time, such as word, which can be typed, spell-checked, and printed at the same time. Within a process, to do multiple tasks at the same time, you need to run multiple "subtasks" at the same time, and we refer to these "subtasks" in the process as threads (thread).

Because each process has at least one thing to do, a process has at least a single thread. Of course, a complex process such as word can have multiple threads, multiple threads can execute simultaneously, multithreading is performed the same way as multiple processes, and the operating system quickly switches between multiple threads, allowing each thread to run briefly alternately, seemingly as if it were executing concurrently. Of course, a multi-core CPU is required to actually execute multiple threads at the same time.

All of the Python programs we wrote earlier are those that perform single-task processes, that is, only one thread. What if we want to do multiple tasks at the same time?

There are two types of solutions:

One is to start multiple processes, although each process has only one thread, but multiple processes can perform multiple tasks in one piece.

Another way is to start a process that starts multiple threads within a process, so that multiple threads can perform multiple tasks in one piece.

Of course, there is a third way, that is, to start multiple processes, each process to start more than one thread, so that the simultaneous execution of more tasks, of course, this model is more complex, and rarely used.

To summarize, there are 3 ways to implement a multitasking:

• Multi-process mode;
• multithreaded mode;
• Multi-process + multithreaded mode.

Performing multiple tasks at the same time is usually not unrelated to each task, but requires communication and coordination with each other, sometimes task 1 must pause waiting for task 2 to finish before it can continue, and sometimes task 3 and task 4 cannot be executed at the same time, so The complexity of multi-process and multi-threaded programs is much higher than the one-process single-threaded program we wrote earlier.

Because the complexity is high, debugging is difficult, so, is not forced, we do not want to write multi-tasking. However, there are many times when there is no more task to do. If you want to see a movie on your computer, you have to play the video by one thread, another thread plays the audio, otherwise, the single-threaded implementation can only play the video before playing the audio, or play the audio before playing the video, which is obviously not possible.

Python supports both multi-process and multi-threaded.

A thread is the smallest execution unit, and a process consists of at least one thread. How to schedule processes and threads is entirely up to the operating system, and the program itself cannot decide when to execute and how long it takes to execute.

Multi-process and multi-threaded programs involve synchronization, data sharing problems, and are more complex to write.

One, multi-process

• To implement multi-process across platforms, you can use multiprocessing the process class in the module
  

If you are going to write a multi-process service program, Unix/linux is undoubtedly the right choice. Because Windows didn't fork call, wouldn't it be possible to write multi-process programs in Python on Windows? Because Python is cross-platform, nature should also provide a cross-platform, multi-process support. multiprocessingmodules are multi-process modules with cross-platform versions. multiprocessing The module provides a Process class to represent a process object.

• • When you create a child process, you only need to pass in an argument (the tuple type) that executes the function and function, creating an Process instance.
  • start() method starts so that the process fork() is simpler to create.
  • join() method can wait for the child process to finish before continuing to run down , typically for inter-process synchronization.

 fromMultiprocessingImportProcessImportOSdefRunproc (name):Print('This is the child process-%s:%s'%(Name,os.getpid ()))if __name__=='__main__':    Print('Parent Process name:%s'%os.getpid ()) P= Process (target=runproc,args= (' Child',)) Print('Child process would start') P.start () P.join ()Print('Child process is ended')
# Parent Process name:3748
# child process would start
# This is a child process-child:8896
# Child process is ended

• If you want to start a large number of child processes, you can use the process pool to batch create child processes, using the pool class

Parallel operations can save a lot of time when using Python for system administration, especially when working with multiple file directories or remotely controlling multiple hosts. If the number of objects in the operation is small, you can also use the process class to dynamically generate multiple processes, more than 10 is OK, but if hundreds or more, then manually to limit the number of processes is particularly cumbersome, the process pool comes in handy.
The pool class can provide a specified number of processes for the user to call, and when a new request is submitted to the pools, a new process will be created to execute the request if it is not full. If the pool is full, the request tells you to wait until the process ends in the pool before a new process is created to execute the requests.

• • Pool (N): Indicates how many child processes are executed at the same time
  • Apply_async (func[, args= () [, kwds={}[, Callback=none]]): It is non-blocking and supports results returned for callbacks
  • Close (): Close the process pool so that it no longer accepts new tasks
  • Terminate (): End worker process, not processing unhandled task
  • Join (): The main process is blocked, waiting for the child process to complete. The join () method must be used after the close () method or the Terminate () method.

 fromMultiprocessingImportPoolImportOS, time, RandomdefLong_time_task (name):Print('Run Task%s (%s) ...'%(name, Os.getpid ())) Start=time.time () time.sleep (Random.random ()* 3) End=time.time ()Print('My Parent PID is:%s'%os.getppid ())Print('Task%s runs%0.2f seconds.'% (name, (End-start )))if __name__=='__main__':    Print('Parent process PID:%s.'%os.getpid ()) P= Pool (4)     forIinchRange (5): P.apply_async (long_time_task, args=(i,))Print('Time :', Time.time ())Print('waiting-subprocesses done ...')    Print('Parent Time:', Time.time ()) P.close () P.join ( )Print('All subprocesses is done .')    #Parent process pid:23444.#time:1501498489.5833461#time:1501498489.5833461#time:1501498489.5833461#time:1501498489.5833461#time:1501498489.5833461#waiting-subprocesses done ...#Parent time:1501498489.5843477#Run Task 0 (18564) ...#Run Task 1 (23532) ...#Run Task 2 (22656) ...#Run Task 3 (23540) ...#My Parent PID is:23444#Task 2 runs 0.24 seconds.#Run Task 4 (22656) ...#My Parent PID is:23444#Task 4 runs 0.50 seconds.#My Parent PID is:23444#Task 1 runs 1.26 seconds.#My Parent PID is:23444#Task 0 runs 2.30 seconds.#My Parent PID is:23444#Task 3 runs 2.60 seconds.#All subprocesses is done .

After a process creates a child process, the relationship between the process and the resulting process is a parent-child relationship, each of which becomes a process and a child process. Once a child process is generated, it executes concurrently with your process, and the child processes share the parent and child processes. The child process executes concurrently with your process once it is generated, and the child process shares the body segment of the parent process and the file that is already open. The sequence of parent and child processes is dispatched by the system. This is why you first print the amount outside of the for loop. Pool (n) is a process pool that generates a concurrency of up to n sub-processes, and if this is changed to 3, then only the first three tasks are allowed to execute and then executed before execution is done.

Python Processes and Threads