Do not use a forced method to kill the python thread or force the python thread

Do not use a forced method to kill the python thread or force the python thread

Preface:

Do not try to use a forced method to kill a python thread, Which is irrational in terms of service design. Multithreading is used for collaborative concurrency of tasks. If you use forcible means to kill threads, there is a high probability of unexpected bugs. Remember that the lock resource will not be released because the thread exits!

We can give two common examples:

1. A thread A gets the lock because it is forced to get rid of it and fails to release the lock resource in A timely manner. As A result, all the threads get the resource and are blocked, this is a typical deadlock scenario.

2. in a common production consumer scenario, a consumer obtains a task from the task queue, but does not drop the ongoing task back to the queue after the task is killed. This causes data loss.

The following describes how to terminate a thread in java and python:

Java has three methods to terminate a thread:

1. Use the exit flag to exit the thread normally, that is, when the run method is complete, the thread is terminated.
2. Use the stop method to forcibly terminate the thread (this is not recommended because the stop method is the same as suspend and resume, and unexpected results may also occur ).
3. Use the interrupt method to interrupt the thread.

Python can be used in two ways:

1. Exit tag
2. Use ctypes to forcibly kill the thread

In both python and java environments, the ideal way to stop and exit a thread is to let the thread stop itself. The so-called thread suicide is to give it a flag, and he exits the thread.

Below we will use multiple methods to test the exception of stopping the python thread. We can view all the execution threads of a process. The process is over-controlled resources, and the thread is used as the scheduling unit. to schedule a process, there must be a thread, the default thread and process pid are the same.

ps -mp 31449 -o THREAD,tid USER   %CPU PRI SCNT WCHAN USER SYSTEM  TIDroot   0.0  -  - -     -   -   -root   0.0 19  - poll_s  -   - 31449root   0.0 19  - poll_s  -   - 31450

After obtaining all the threads of the process, we learned through strace that 31450 is the thread id that requires us to kill. When we kill, the whole process will crash. In a multi-threaded environment, the signal is sent to the whole process. Generally, all threads have the opportunity to receive this signal. The process executes the signal processing function in the context of the thread that receives the signal, it is hard to know which thread to execute. That is to say, the signal will randomly send a thread of the process.

strace -p <span style="font-size:14px;line-height:21px;">31450</span> Process <span style="font-size:14px;line-height:21px;">31450</span> attached - interrupt to quitselect(0, NULL, NULL, NULL, {0, 320326}) = 0 (Timeout)select(0, NULL, NULL, NULL, {1, 0})   = 0 (Timeout)select(0, NULL, NULL, NULL, {1, 0})   = 0 (Timeout)select(0, NULL, NULL, NULL, {1, 0})   = ? ERESTARTNOHAND (To be restarted)--- SIGTERM (Terminated) @ 0 (0) ---Process <span style="font-size:14px;line-height:21px;">31450</span> detached

The above problems are actually consistent with the pthread description. When we add the signal processing function to the python code, the callback function can prevent the entire process from exiting. The problem arises. The signal function cannot identify which thread you want to kill, that is, A thread cannot be killed accurately. Although you send the signal to the 31450 thread id, the signal receiver is the process owner. In addition, the parameters passed to the signal processing function are only the number of signals and the signal stack, which are dispensable.

After signal processing is added, the process will not exit

select(0, NULL, NULL, NULL, {1, 0})   = 0 (Timeout)select(0, NULL, NULL, NULL, {1, 0})   = ? ERESTARTNOHAND (To be restarted)--- SIGTERM (Terminated) @ 0 (0) ---rt_sigreturn(0xffffffff)        = -1 EINTR (Interrupted system call)select(0, NULL, NULL, NULL, {1, 0})   = 0 (Timeout)select(0, NULL, NULL, NULL, {1, 0})   = 0 (Timeout)

If you want to kill a thread from an external notification, you can build and use the rpc service or communicate in other ways. The signal is not acceptable because more information cannot be transmitted.

The python thread is not a simulated, but a real kernel thread. The kernel calls the pthread method, but the Python upper layer does not provide a way to close the thread, which requires our own grasp. We strongly recommend that you use the event or custom flag bit method. If you want to force kill the thread, you can use the python ctypes PyThreadState SetAsyncExc method to force exit. This will not affect the running python service.

The implementation principle of this function is relatively simple. In fact, it is also a mark bit in the python virtual machine, and then the virtual machine runs an exception to cancel the thread, the virtual opportunity to help you do a try cache. Do not kill a python thread externally. Although you can find the thread id through ctypes, kill the whole process.

The following code is an example of killing a thread with ctypes. It is not recommended because it is too rough.

import ctypes def terminate_thread(thread):  if not thread.isAlive():    return   exc = ctypes.py_object(SystemExit)  res = ctypes.pythonapi.PyThreadState_SetAsyncExc(    ctypes.c_long(thread.ident), exc)  if res == 0:    raise ValueError("nonexistent thread id")  elif res > 1:    ctypes.pythonapi.PyThreadState_SetAsyncExc(thread.ident, None)    raise SystemError("PyThreadState_SetAsyncExc failed")

Let's take a look at the PyThreadState source code. In short, the thread exception mode is triggered. If you are interested, you can read the python pystate. c design and share some videos with youtube.

IntPyThreadState_SetAsyncExc (long id, PyObject * exc) {PyInterpreterState * interp = GET_INTERP_STATE ();... HEAD_LOCK (); for (p = interp-> tstate_head; p! = NULL; p = p-> next) {if (p-> thread_id = id) {find the thread id from the linked list to avoid deadlocks. We need to release head_mutex. PyObject * old_exc = p-> async_exc; Py_XINCREF (exc); # add the number of references to this object p-> async_exc = exc; # Use the exc mode HEAD_UNLOCK (); py_XDECREF (old_exc... return 1; # thread destroyed successfully} HEAD_UNLOCK (); return 0 ;}

Native posix pthread can use ptread_cancel (tid) to end the subthread in the main thread. However, the Python thread library does not support this. The reason is that we should not forcibly end a thread, which brings many risks and should end the thread itself. Therefore, in Python, the recommended method is to cyclically judge a flag in the Child thread and change the flag in the main thread. When the child thread reads the flag changes, it ends itself.

Similar to this logic:

def consumer_threading(): t1_stop= threading.Event() t1 = threading.Thread(target=thread1, args=(1, t1_stop))  t2_stop = threading.Event() t2 = threading.Thread(target=thread2, args=(2, t2_stop))  time.sleep(duration) #stop the thread2 t2_stop.set() def thread1(arg1, stop_event): while(not stop_event.is_set()):   #similar to time.sleep()   stop_event.wait(time)   pass  def thread2(arg1, stop_event): while(not stop_event.is_set()):   stop_event.wait(time)   pass

In a simple summary, although we can use pystats in ctypes to control threads, this method of brute force thread interruption is unreasonable. Select the suicide mode! What if your thread is experiencing io blocking and cannot judge the event? Your program needs to be optimized. At least active timeouts are required on the network I/O layer to avoid continuous blocking.