Python Thread (threading) session (multiprocessing)

Python Thread (threading) session (multiprocessing)

Recently learned two Python library, one is responsible for managing the thread, a management process, the original has been written a few single-threaded programs, although the web is also about concurrency and more involved in threading, but are framework management, learning > Later it was found that understanding threads and processes also helped Python's web development. Let's talk about the application of these two libraries to Python's support for threads and processes.

Python 's support for threads is not very good, so you can criticize the drawbacks of Python's multithreading in many articles, but why is python not good for multithreaded support, why other languages such as C, Java, C # Static language does not have this disadvantage.

First of all we need to know that Python is an explanatory language, each piece of code requires an interpreter to compile and run, the interpreter has many of the most important is CPython, and other IronPython and Jython, The official is the CPython interpreter, we generally say that multithreading support is not good for the CPython interpreter (the most people to omit the Python interpreter), why the Python interpreter for multithreading support is not good, because the Gil exists, Of course, this existence is due to the nature of the language.

What's the Gil, and here's the official explanation?

In CPython, the global interpreter lock, or GIL, was a mutex that prevents multiple native threads from executing Python by Tecodes at once. This lock is necessary mainly because CPython ' s memory management are not thread-safe. (However, since the GIL exists, other features has grown to depend on the guarantees that it enforces.)

The Gil is a python mutex, and the simple understanding is that the code will lock the Python interpreter. Understanding what the code is locked in must first understand what multithreading is

Multithreading represents a main thread, multiple child threads, the main thread is the program executes when the system automatically gives you a thread to apply, and the child threads we can understand as a block of code, we can take full advantage of hardware support such as multicore, let one CPU executes the main thread, the other CPU executes the sub-thread, Through the operating system of virtual memory technology to allow all threads to share the same code space to improve the efficiency of the code, we can easily compare a process to a train, the car head for the main thread, each car is a sub-thread, as long as your car (sub-thread) more, you shipped more goods, but also to consider the aspect of hardware,

Understand what multithreading is, we can explain the Gil to multi-core CPU performance impact, in a single-core CPU, the main thread in the release of the Gil, the CPU to the child thread, sub-thread code block get Gil, and then execute, so that can fully utilize the CPU, This Gil has no impact on the performance of single-core, can get 100% of the use, but in the multi-core of the time there is a problem, if the main thread of the code has always needed an interpreter to execute, such as the following

    GIL.acquire()    try:        while True:            do_something()    finally:        GIL.release()

The main thread code to the Gil lock and unlock only a small interval of a system time, sub-thread in the other CPU core to get the Gil untie the CPU after the dispatch command to be awakened, but when awakened, the main thread of the code again locked Gil, and then only wait for the main thread next dispatch command, But to switch time and switch back to the adjustment state, has been in the wake, waiting for the vicious circle, multi-core function is not played out and more than the single core is worse, so python because the existence of Gil to the dense thread support is poor, but if the main thread is in the execution of the web to wait for user input, Instead of using the interpreter to execute code every minute and every second, the benefits of multithreading can be played out.

Solution Solutions

Gil as a bug in the interpreter's existence, we also have a certain solution, the thread, and with CType bypass the interpreter is our general solution, you want to learn more can see this next major Jie Shao with multiprocessing to bypass the bottleneck of multithreading

Wire lock and process lock in order to achieve thread safety, we also need to use the existence of locks, we first used the following code to verify the multithreading for thread safety issues. We declare a thread lock threading.Lock(),

  class Counter (object): Def __init__ (self, start=0): Self.lock = Threading.        Lock () Self.value = Startdef increment (self): Logging.debug (' Waiting for Lock ') Self.lock.acquire () Try: If Self.value < 8:time.sleep (2) # Analog load Logging.debug (' acquired lock ') Self.value =        Self.value + 1 finally:self.lock.release () def worker (c): For I in range (2): pause = Random.random () Logging.debug (' Sleeping%0.02f ', pause) time.sleep (pause) c.increment () logging.debug (' Done ') counte R = Counter () for I in range: T = Threading. Thread (target=worker,args= (counter,)) T.start () Main_thread = Threading.currentthread () for T in Threading.enumerate () : If T is not main_thread:t.join () # Protection thread Logging.debug (' counter:%d ', counter.value) #得到value值  

We get a value of 8 after we run counter.value it, which is good to understand because we limit its size to less than 8 o'clock to increase by 1, but if we remove the lock, we self.lock.acquire() self.lock.release() comment out the result is a 21, and the result of each run may be different, Because threads have a certain amount of time () when they implement self-increment, when time.sleep(2) multiple processes execute when they fetch a value of 7 o'clock from the stack, counter.value they all meet counter.value less than 8, so they all perform the self-increment, between the system load of 2 seconds ( time.sleep(2) The number of threads executing will escape the restrictions we give him, which creates a thread that is unsafe, but after we add a lock to him, no matter how many threads are opened, the end result is 8. In Python we can be seen as the same thing as our lock and process locks.

PS: When the same thread fights each other for a lock, the failed incoming thread queue waits for the lock to unravel. Thread process mode of work line

A single line is primarily implemented by locks, threads create locks by locking threading.Lock() objects, process multiprocessing.Lock() locks are created by objects, and single-line operations are generally a protection against shared data modifications.

Parallel

Parallel operation is generally a kind of data sharing, generally do not involve modification of public data, we can create a lot of threads and processes in parallel operation, can also limit the number of threads and processes in parallel, the two choices are mainly to determine whether the code type is I/O dense or thread-intensive. How to limit the number of concurrent quantities we can pass threading.Semaphore（sizenum） (process for multiprocessing.Semaphore(sizenum) ) we can control the number of threads to share. The process provides a type of process pool ( multiprocessing.Pool ), and we can create a process pool that maintains a certain range of processes, but he has no control over the number of concurrent processes, just to help us create this process pool, where each process does not only perform one task, but may execute multiple methods through a process.

Single-line hybrid parallel

Single-line and parallel blending we can do this by setting the lock in the code, of course, Python gives us two objects to implement single-line and parallel control, the thread is threading.Event() and threading.Condition() , the process is multiprocessing.Event() and multiprocessing.Condition() two objects are provided with a command instruction, But the event object can be used to determine whether the command is released and react accordingly, whereas the condition object prefers to perform parallel operations when the command is released.

How threads and processes communicate

When we want threads and processes to perform some fixed tasks together, we need to be able to communicate between threads and processes, thread and process communication we use Queues ( Queue ), a bit of a difference between processes and threads, that the Queue Queue objects passed by the process must be pickle, and in order to be able to use join()(Protection process) task_done (Notification task complete), we generally use the JoinableQueue substitution Queue in the process.

The queue objects pass through put and get communicate, we put the task put up, Queue automatically assigned to the current thread or process, so that the task can be implemented in the assembly-flow session.

Reference

12/26/2015 10:50:21 PM Gil Wiki Info

Gil Blog

Python Thread (threading) session (multiprocessing)