Python rookie Diary 8

1. The multi-process programming concept in Python: Because Python is cross-platform, nature should also provide a cross-platform, multi-process support. The multiprocessing module is a multi-process module with cross-platform versions. The multiprocessing module provides a process class to represent a processing object.

2. Multi-threaded programming in Python: when we introduce the multiprocessing module, we create a process for the function that needs to be called: b= process (target= function name, args= (argument)), because process is a class, So B is the process object of the target function, and we can also add the Name property inside to define the process, but Python usually gives the process a name automatically, B.start () executes the function by invoking the start () function, P.join () calls the Join () function to show that when the call to execute function calls is complete immediately after the execution of the following statement code, the function of the left and right is like a undertake function.

3. Process pool in Python: If you need to pass in multiple different arguments to a function, you can create multiple child processes through the process pools. Specific practices: The From multiprocessing Import pool first introduces the process pool, b= pool (n) to create a process pool object, the n means that can be simultaneously n processes, it is important to remember the multi-process call, the process is alternately executed, b.apply_ Async (function name, args= (argument)) creates a process, calls Apply_async () multiple times to create several sub-processes, P.close () completes a process by calling close () after execution of a multi-process, and indicates that a new process cannot continue to be added, P.join () to undertake the role.

4. There is certainly a need for communication between processes, and the operating system provides a number of mechanisms to enable interprocess communication. Python's multiprocessing module wraps the underlying mechanism, provides a queue, pipes and many other ways to exchange data, in the case of queue, we can write two functions, one is the read function, one is the write function, after the introduction of the queue, We B=queue () to create a B queue object and then use a= in the Read function B.get (True) to remove the value from queue B and write value to queue B through B.put (value) in the Write function, so that we can then create two processes and then start () call execution, between the two processes to communicate with the queue.

5. Under Unix/linux, you can use the fork () call to implement multiple processes. To achieve multi-process across platforms, you can use the multiprocessing module. Inter-process communication is realized through queue, pipes and so on.

6. Multithreaded programming in Python: Python's standard library provides two modules: _thread and Threading,_thread are low-level modules, threading is an advanced module that encapsulates _thread. In most cases, we only need to use the Advanced module threading

7. Multi-process programming practices in Python: Similar to multi-process programming, b= Threading. Thread (target= function name, name= thread alias), T.start () Start thread execution function, T.join () take effect

8. Multi-threaded and multi-process the biggest difference is that, in a multi-process, the same variable, each has a copy in each process, do not affect each other, and in multiple threads, all variables are shared by all threads, so any one of the variables can be modified by any one thread, therefore, The biggest danger of sharing data between threads is that multiple threads change a variable at the same time, and the content is scrambled. (The variables here refer to the global variables in the module)

9. lock,b= threading in multi-threading. Lock () to create the lock object B, for the function in relation to the global variable to define the lock mechanism lock.acquire (), that is, multithreading in the execution of this function must first get this lock, and then to the following statement code, or enter the queue to wait for, After executing the function, Lock.release () releases the lock and continues execution to the thread in the wait queue, thus avoiding the instability of the global variable when the thread is alternately executing.

10. Multithreaded programming, the model is complex, prone to conflict, must be isolated with locks, but also beware of the occurrence of deadlocks. The Python interpreter was designed with a Gil global lock, resulting in multi-threaded inability to take advantage of multicore. Multi-threaded concurrency is a beautiful dream in Python.

11. In a multithreaded environment, each thread has its own data. It is better for a thread to use its own local variables than to use global variables, because local variables can only be seen by the thread themselves and not affect other threads, and changes to global variables must be locked. Before we had to lock the global variables in multi-threading with lock, Python provided another way to not lock the global variables, the "global variables" of a single line thread, the variables that all the functions of the line thread could access, but the other threads could not access ... It's called threadlocal.

How to use 12.Threadloca: we create a b global variable by b= threading.local (), and B is a Threadlocal object that can be defined B.A,B.C in each function. B.E The local variables in these threads are the properties of B, as if B is the global variable of the thread, and the latter B.A, B.E is a thread's global variables, we can think of them all as a global variable of a thread, so that when multithreading is executed alternately, each thread can use its own global variables, avoiding the problem of sharing the global variables in the source program.

13. First, to achieve multi-tasking, usually we will design the Master-worker mode, Master is responsible for assigning tasks, the worker is responsible for performing the task, therefore, in a multitasking environment, usually a master, multiple workers. If you implement master-worker with multiple processes, the master process is master and the other process is Worker. If the Master-worker is implemented with multithreading, the main thread is master, and the other threads are worker.

14. Multi-process advantages and disadvantages: The greatest advantage of multi-process mode is high stability, because one child process crashes, does not affect the main process and other child processes. (Of course, the main process hangs up all the processes, but the master process is only responsible for assigning tasks, the probability of hanging off is low)
The disadvantage of the multi-process mode is that the cost of creating the process is large, and under the Unix/linux system, it is OK to make a fork call, and the process overhead under Windows is huge. In addition, the operating system can run simultaneously the number of processes is also limited, under memory and CPU constraints, if there are thousands of processes running simultaneously, the operating system even scheduling will be a problem

15. Multithreading pros and Cons: Multithreaded mode is usually faster than a multi-process, but also quickly not go anywhere, and, the multi-threaded mode of the fatal disadvantage is that any one thread hangs can directly cause the entire process to crash, because all threads share the memory of the process. On Windows, if a thread executes a problem with the code, you can often see the hint that "the program is performing an illegal operation that is about to close," which is often a problem with a thread, but the operating system forces the entire process to end. Under Windows, multithreading is more efficient than many processes.

16. Whether it is multi-process or multi-threading, as long as the number of a lot, efficiency certainly not go, why? Because multithreaded multi-process is executed alternately, there will be a switchover process, although the switching process is very fast, but it also takes time. If there are thousands of tasks at the same time, the operating system may be mainly busy switching tasks, there is not much time to perform the task, the most common is the hard drive, the point window unresponsive, the system is in suspended animation state. Therefore, once the multi-tasking to a limit, it will consume all the resources of the system, resulting in a sharp decline in efficiency, all tasks are not good.

17. Compute-intensive tasks: compute-intensive tasks are characterized by a large number of calculations, consuming CPU resources, such as the calculation of PI, HD decoding video, and so on, all rely on the CPU's computing power. This computationally intensive task can be accomplished with multitasking, but the more tasks, the more time it takes to switch tasks, the less efficient the CPU is to perform the task, so the most efficient use of the CPU should be equal to the number of cores in the CPU. Compute-intensive tasks are critical to the efficiency of your code because they consume CPU resources primarily. Scripting languages like Python are inefficient and are completely unsuitable for compute-intensive tasks. For computationally intensive tasks, it is best to write in C.

IO-intensive tasks: tasks involving networking and disk IO are IO-intensive tasks characterized by low CPU consumption and most of the time the task is waiting for the IO operation to complete (because IO is much slower than CPU and memory). For IO-intensive tasks, the more tasks you have, the higher the CPU efficiency, but there is a limit. Most of the tasks that are common are IO-intensive tasks, such as Web applications. IO-intensive task execution, 99% of the time spent on the IO, the time spent on the CPU is very small, so the fast-running C language to replace the very low-speed scripting language with Python, completely unable to improve operational efficiency. For IO-intensive tasks, the most appropriate language is the one with the most efficient (least code) language, the scripting language is preferred, the C language is the worst


if __name__ = = ' __main__ ': The function is to determine whether the module is executed directly or is being invoked to another module for execution,
If we are directly executing a. py file, the file then "__name__ = = ' __main__ '" is true, but if we import the file through import from another. py file, then __name__ The value is the name of our py file, not the __main__. This feature also has a use: When debugging code, in "if __name__ = = ' __main__ '" to add some of our debugging code, we can let the external module
When the call does not execute our debugging code, but if we want to troubleshoot the problem, directly execute the module file, debugging code can run properly!

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Python rookie Diary 8