Python Lesson 5-custom thread pool and python Thread Pool

Python Lesson 5-custom thread pool and python Thread Pool
Summary:

1. low version Thread Pool

2. Out-of-print Thread Pool

 

1. low version Thread Pool

Design Concept: Using queue

Put the Thread class name in the queue and execute one to get it.

1 import queue 2 import threading 3 4 5 class ThreadPool (object): 6 7 def _ init _ (self, max_num = 20): 8 self. queue = queue. queue (max_num) # create a Queue with a maximum of 20 9 for I in range (max_num): 10 self. queue. put (threading. thread) # Put the class name in the queue 11 12 def get_thread (self): 13 return self. queue. get () # retrieve the class name from the queue 14 15 def add_thread (self): 16 self. queue. put (threading. thread) # enter the class name into the queue 17 18 def func (arg, p): # define a function 19 print (arg) 20 import time21 time. sleep (2) 22 p. add_thread () 23 24 25 pool = ThreadPool (10) # create an object and execute the constructor of this class. Put the Thread class name in the queue 26 27 for I in range (30): 28 thread = pool. get_thread () # Call the get_thread method of the object and retrieve the class name 29 t = thread (target = func, args = (I, pool) # create an object and execute func, the parameter is 30 t in args. start ()

Because this method requires the user to modify the original function, PASS Parameters in the original function, and call the method has also changed, and there are Idle threads that waste resources, the actual operation is not convenient, therefore, the thread pool of the next version is designed.

 

2. Out-of-print Thread Pool

Design Concept: Using queue

A. Put tasks in the queue

B. The thread retrieves the task again and again, and obtains the task as soon as the thread is idle.

1 import queue 2 import threading 3 import contextlib 4 import time 5 6 StopEvent = object () 7 8 9 class ThreadPool (object): 10 11 def _ init _ (self, max_num, max_task_num = None): 12 if max_task_num: 13 self. q = queue. queue (max_task_num) 14 else: 15 self. q = queue. queue () 16 self. max_num = max_num 17 self. cancel = False 18 self. terminal = False 19 self. generate_list = [] 20 self. free_list = [] 21 22 def run (self, func, args, callback = None): 23 "" 24 thread pool executes a task 25: param func: task function 26: param args: required parameter 27: param callback: the callback function executed when the task fails or succeeds. The callback function has two parameters: 1. The execution status of the task function; 2. return Value of the task function (default value: None, that is, do not execute the callback function) 28: return: if the thread pool has been terminated, True is returned; otherwise, None 29 "30 if self. cancel: 31 return 32 if len (self. free_list) = 0 and len (self. generate_list) <self. max_num: 33 self. generate_thread () 34 w = (func, args, callback,) 35 self. q. put (w) 36 37 def generate_thread (self): 38 "39 create a thread 40" 41 t = threading. thread (target = self. call) 42 t. start () 43 44 def call (self): 45 "46 loop to get the task function and execute the task function 47" 48 current_thread = threading. currentThread () 49 self. generate_list.append (current_thread) 50 51 event = self. q. get () 52 while event! = StopEvent: 53 54 func, args, callback = event 55 try: 56 result = func (* args) 57 success = True 58 failed t Exception as e: 59 success = False 60 result = None 61 62 if callback is not None: 63 try: 64 callback (success, result) 65 failed t Exception as e: 66 pass 67 68 with self. worker_state (self. free_list, current_thread): 69 if self. terminal: 70 event = StopEvent 71 else: 72 event = self. q. get () 73 else: 74 75 self. generate_list.remove (current_thread) 76 77 def close (self): 78 "79 after all tasks are executed, all threads stop 80" 81 self. cancel = True 82 count = len (self. generate_list) 83 while count: 84 self. q. put (StopEvent) 85 count-= 1 86 87 def terminate (self): 88 "89 terminate the thread 90" 91 self, regardless of whether there are tasks. terminal = True 92 93 while self. generate_list: 94 self. q. put (StopEvent) 95 96 self. q. queue. clear () 97 98 @ contextlib. contextmanager 99 def worker_state (self, state_list, worker_thread): 100 "" 101 is used to record the number of threads waiting in the thread: 102 "" 103 state_list.append (worker_thread) 104 try: 105 yield106 finally: 107 state_list.remove (worker_thread) 108 109 110 111 # How to use112 113 pool = ThreadPool (5) 114 115 def callback (status, result): 116 # status, execute action status117 # result, execute action return value118 pass119 120 def action (I): 121 print (I) 122 123 for I in range (30): 124 ret = pool. run (action, (I,), callback) 125 126 time. sleep (3) 127 print (len (pool. generate_list), len (pool. free_list) 128 print (len (pool. generate_list), len (pool. free_list) 129 pool. close () 130 # pool. terminate ()