Python threading Parent process does not die, child threads do not exit: How can I recycle threads after I have finished using threads?

Recently, multiple threads in Python were used for concurrent ping operations, where In_queue and out_queue two blocking queues were used to ensure thread safety. Found a problem,

Is that the main process of creating multi-threaded is not to quit, it (the main process) created by all the threads will not be killed, only the state is sleeping state, so there is a problem, the Linux system assigned to each user the maximum number of open processes is limited, If a process like flask or Django does not exit theoretically, it will create more and more threads, sooner or later, so that the system cannot allocate any resources to the user ....

The code is as follows:

#!/usr/bin/env python
#-*- coding:utf-8 -*-

from threading import Thread 
import subprocess 
from Queue import Queue, Empty
import re 
import sys,os,time 
import platform 

def ping(
    hostname = None,
    count = 1,
    timeout = 3
):
    data = {
        ‘errorcode‘:0xff,
        ‘errorinfo‘:None,
        ‘data‘:{
            ‘hostname‘:hostname,
            ‘isUp‘:None
        }
    }
    try:
        outfile = ‘‘
        if "Linux" == platform.system():
            cmd = "ping -c %d -w %d %s"%(count,timeout,hostname)
            outfile = "/dev/null"
        elif "Windows" == platform.system():
            cmd = "ping -n %d -w %d %s"%(count,timeout,hostname)
            outfile = "ping.temp"
        ret = subprocess.call(cmd, shell=True, stdout=open(outfile,‘w‘), stderr=subprocess.STDOUT)
        data[‘data‘][‘isUp‘] = True if 0 == ret else False
        data[‘errorcode‘] = 0x00
    except Exception,e:
        print traceback.format_exc()
    finally:
        return data
        
        
        
def ping_with_queue(in_queue, out_queue):
    while True:
        ip = in_queue.get()
        data = ping(hostname = ip)
        out_queue.put(data)
        in_queue.task_done()#向任务已完成的队列,发送一个通知信号


def append_result(out_queue, result):
    while True:
        data = out_queue.get()
        result.append(data)
        out_queue.task_done()#向任务已完成的队列,发送一个通知信号

        
        
def get_all_device_status(ips):
    data = {
        ‘errorcode‘:0xff,
        ‘errorinfo‘:None,
        ‘data‘:[]
    }
    result = []
    threads_nums = len(ips)
    in_queue = Queue()
    out_queue = Queue()
    for i in range(threads_nums):
        t = Thread(target=ping_with_queue, args=(in_queue,out_queue))
        t.setDaemon(True)
        t.start()
        
    for ip in ips:
        in_queue.put(ip)

    for i in range(threads_nums):
        t = Thread(target=append_result, args=(out_queue,data[‘data‘]))
        t.setDaemon(True)
        t.start()
    data[‘errorcode‘] = 0x00
    in_queue.join()
    out_queue.join()
    return data

if __name__ == ‘__main__‘:
    ips = [ ‘10.1.31.‘+str(i) for i in xrange(0xff)]
    print get_all_device_status(ips)
    print ‘main process begin sleep 20 seconds....‘
    time.sleep(20)
    print ‘main process exit‘

	

The running process is explained as follows:

2. The program is not running, the system's thread status is as follows: The system currently has a total of 592 threads, of which 1 are running, others are in the sleep state 650) this.width=650; "Src=" https://s1.51cto.com/top-h Wyfs02/m01/8d/f8/wkiol1ixjmdx2qyiaajxqwjelre890.jpg "title=" system current thread Run State "alt=" Wkiol1ixjmdx2qyiaajxqwjelre890.jpg " />

4. Running the system, but the multi-threaded task has not completed, In_queue and Out_queue queue is still blocked state, top-h display as follows: At this time there are 1480 sites in the system, that is to say my program creates 1480-592 +1= 889 threads, in theory will create 255* 2= 510 threads, as for the more create more out of 889-510 = 379 threads, I guess is subprocescess call the System ping command first off, again do not consider the situation of the 379 threads, only consider the whereabouts of 510 threads ... But this is not the point, this stage, two queues are still in a blocking state, the creation of multithreading has not all returned 650) this.width=650; "Src=" https://s3.51cto.com/wyfs02/M00/8D/F8/ Wkiol1ixjrmhdbdgaajs8e29ruw042.jpg "title=" is creating a thread in the ping task ... "alt=" wkiol1ixjrmhdbdgaajs8e29ruw042.jpg "/>

6. In_queue and out_queue two queues are not blocking the main process, multi-threaded task execution completes, get_all_device_status (IPS) already has the return value, the main process sleeps 20 seconds, the shell displays as follows: 650) this.width=650, "src=" Https://s4.51cto.com/wyfs02/M02/8D/F8/wKioL1ixkwTzQar3AATC_HWbwSE831.jpg "title=" The queue does not block the main process again, the task is completed, and the ping result of all devices has been taken "alt=" wkiol1ixkwtzqar3aatc_hwbwse831.jpg "/> Main process because Time.sleep (20) enters 20 seconds of sleep, At this point top-h is shown as follows: In_queue and Out_queue no longer block the main process, the task also has a return value, but shows a total of 1101 threads, that is, the remaining 1101-592+1 = 510 that is, subprocess that 30 multi-brother thread has exited, But 510 threads created by the program did not exit ... 650) this.width=650; "src=" Https://s5.51cto.com/wyfs02/M02/8D/F8/wKioL1ixlPPDe-b7AAJN3DdHW6Q853.jpg "title=" The queue no longer blocks the main process, all threads run back, but do not exit .... "alt=" wkiol1ixlppde-b7aajn3ddhw6q853.jpg "/>

8. The main process sleeps 20 seconds, exits, the program runs completed, live into the exit, it created all the threads have exited, the current system thread number is: 590, that is, 1101-590+1 = 512 threads, after the main process exits after exiting ... top-h display as follows: 650) this.width=650; "src=" Https://s1.51cto.com/wyfs02/M00/8D/FA/wKiom1ixlqaS6NzMAAJF2W77DrU819.jpg "title=" after the main process exits, The thread it creates also exits the "alt=" wkiom1ixlqas6nzmaajf2w77dru819.jpg "/>" That's the problem, if the main process sleeps longer, like a year, then within a year, all the threads will not quit, and, earlier, If you run the program repeatedly, you cannot quit the 510 threads you created .... This will sooner or later allocate the operating system to the current user's maximum number of threads run out (ulimit-n) ... By the time, all actions of the user cannot be used ....

10. Summary, according to my current code, is the thread after the completion of the task, or will exist, state programming sleeping state, only the main process exit to exit .... But if you use flask or Django to invoke multithreading, because Flask and Django and so easily do not quit ... So there's the problem I'm talking about ... So, how can you use the Threading Queue task to have a return after the main process kills the created thread ...?

This article is from the "Mr_computer" blog, make sure to keep this source http://caochun.blog.51cto.com/4497308/1901269

Python threading Parent process does not die, child threads do not exit: How can I recycle threads after I have finished using threads?