Code of the thread pool instance implemented by python, And the python thread pool instance

Code of the thread pool instance implemented by python, And the python thread pool instance

The python3 standard library comes with the thread pool ThreadPoolExecutor and the process pool ProcessPoolExecutor.

If you are using python2, you can download a module called threadpool, which is a thread pool. You can use the multiprocessing. Pool that comes with python for the process Pool.

You can also write a threadpool by yourself.

# Coding: UTF-8 import Queueimport threadingimport sysimport timeimport math class WorkThread (threading. thread): def _ init _ (self, task_queue): threading. thread. _ init _ (self) self. setDaemon (True) self. task_queue = task_queue self. start () self. idle = True def run (self): sleep_time = 0.01 # rest for 10 milliseconds when no task is available for 1st times. multiply = 0 while True: try: # Get a task func from the queue, args, kwargs = self. task_queue.get (block = False) self. idle = False multiply = 0 # execution func (* args, ** kwargs) handle T Queue. empty: time. sleep (sleep_time * math. pow (2, multiply) self. idle = True multiply + = 1 continue failed T: print sys. exc_info () raise class ThreadPool: def _ init _ (self, thread_num = 10, max_queue_len = 1000): self. max_queue_len = max_queue_len self. task_queue = Queue. queue (max_queue_len) # The task waits for the Queue self. threads = [] self. _ create_pool (thread_num) def _ create_pool (self, thread_num): for I in xrange (thread_num): thread = WorkThread (self. task_queue) self. threads. append (thread) def add_task (self, func, * args, ** kwargs): ''' Add a task, return the length of the task waiting queue. Before calling this method, call isSafe () to check whether there are many waiting tasks to prevent the submitted tasks from being denied ''' try: self. task_queue.put (func, args, kwargs) does t Queue. full: raise # directly throws an exception when the queue is Full and does not execute return self. task_queue.qsize () def isSafe (self): ''' the number of waiting tasks is far from the alert line ''' return self. task_queue.qsize () <0.9 * self. max_queue_len def wait_for_complete (self): ''' all tasks waiting for submission to the thread pool are completed ''' # The task waits for the queue to become empty while not self. task_queue.empty (): time. sleep (1) # second, so the computing thread must change to the idle state while True: all_idle = True for th in self. threads: if not th. idle: all_idle = False break if all_idle: break else: time. sleep (1) if _ name _ = '_ main _': def foo (a, B): print a + B time. sleep (0.01) thread_pool = ThreadPool (10,100) ''' test fails on Windows, while Queue on Windows. queue is not a thread-safe ''' size = 0 for I in xrange (10000): try: size = thread_pool.add_task (foo, I, 2 * I) Queue t Queue. full: print 'queue full, queue size is ', size time. sleep (2)

Summary

The above is all about the code of the thread pool instance implemented in python. I hope it will be helpful to you. If you are interested, you can continue to refer to other related topics on this site. If you have any shortcomings, please leave a message. Thank you for your support!