Python custom thread pool Implementation Method Analysis, python custom Thread Pool

Python custom thread pool Implementation Method Analysis, python custom Thread Pool

This example describes how to implement a Python custom thread pool. We will share this with you for your reference. The details are as follows:

Concerning the multithreading of python, the existence of GIL has been criticized by many groups, saying that the multithreading of python is not a real multithreading. However, the efficiency of multi-thread processing of IO intensive tasks can be leveraged.

The thread pool I implemented is actually implemented based on the silver horn idea.

Main ideas:

Task acquisition and execution:

1. Add the task to the queue and wait for the thread to obtain and execute the task.
2. Generate threads as needed, and each thread cyclically retrieves tasks.

Thread destruction:

1. When the task is obtained as a Terminator, the thread stops.
2. When the thread pool is closed (), a terminator equivalent to the generated thread is added to the task queue.
3. When the thread pool is terminate (), set the next time the thread receives the task as the Terminator.

Process Outline Design:

Code details:

Import threadingimport contextlibfrom Queue import Queueimport timeclass ThreadPool (object): def _ init _ (self, max_num): self. stopEvent = 0 # thread task Terminator. When a thread obtains StopEvent from the queue, this thread can be destroyed. Can be set to any value different from the task. Self. q = Queue () self. max_num = max_num # maximum number of threads self. terminal = False # whether to set the thread pool to force terminate self. created_list = [] # list of created threads self. free_list = [] # list of Idle threads self. deamon = False # Whether the thread is a background thread def run (self, func, args, callback = None): "" the thread pool executes a task: param func: task function: param args: required parameter of the task function: param callback: return: if the thread pool has been terminated, True is returned. Otherwise, None "" if len (self. free_list) = 0 and len (self. created_list) <self. max_num: se Lf. create_thread () task = (func, args, callback,) self. q. put (task) def create_thread (self): "Create a thread" "t = threading. thread (target = self. call) t. setDaemon (self. deamon) t. start () self. created_list.append (t) # Add the current thread to the list of created threads created_list def call (self): "loop to obtain the task function and execute the task function" current_thread = threading. current_thread () # obtain the current thread object · event = self. q. get () # get the task while event from the task queue! = Self. stopEvent: # determine whether the obtained task is the terminator func, arguments, callback = event # obtain the function name, parameter, and callback function name from the task try: result = func (* arguments) func_excute_status = True # Success t Exception as e: func_excute_status = False result = None print 'function execution error ', e # print the error message if func_excute_status: # The callback function can be executed only after the func execution is successful. if callback is not None: # try: callback (result) to determine whether the callback function is null) failed t Exception as e: print 'callback function execution produces error', e # print error message wi Th self. worker_state (self. free_list, current_thread): # after a task is executed, add the thread to the idle list. Then, continue to fetch the task. if the task is obtained, the thread will be removed from the idle list. if self. terminal: # judge the thread pool termination command. if the thread pool needs to be terminated, the next retrieved task will be StopEvent. Event = self. stopEvent else: # Otherwise, continue to obtain task event = self. q. get () # q. the get () method blocks the thread and keeps waiting for else: # If the task received by the thread is the Terminator, destroy the thread # Remove self from the list of created threads from created_list. created_list.remove (current_thread) def close (self): "After all tasks are executed, all threads stop" full_size = len (self. created_list) # Add a terminator to the thread queue based on the number of created threads. While full_size: self. q. put (self. stopEvent) full_size-= 1 def terminate (self): "terminate a thread regardless of whether there are tasks" self. terminal = True while self. created_list: self. q. put (self. stopEvent) self. q. queue. clear () # clear task queue def join (self): "blocking thread pool context, so that all threads can continue after execution" for t in self. created_list: t. join () @ contextlib. contextmanager # context processor so that it can use the with statement to modify def worker_state (self, state_list, worker_thread): "used to record the number of threads waiting in the thread" state_list.append (worker_thread) try: yield finally: state_list.remove (worker_thread) if _ name _ = '_ main _': def Foo (arg): return arg # time. sleep (0.1) def Bar (res): print res pool = ThreadPool (5) # pool. deamon = True # It must be in the pool. before run, set for I in range (1000): pool. run (func = Foo, args = (I,), callback = Bar) pool. close () pool. join () # pool. terminate () print "number of tasks in the task queue % s" % pool. q. qsize () print "number of currently active sub-threads: % d" % threading. activeCount () print "current thread creation list: % s" % pool. created_list print "current thread creation list: % s" % pool. free_list

Context processing:

Here is a simple example:

The following code manually defines a myopen method to simulate our common with open () as f: Statement. The specific use of the contextlib module will be opened separately.

# Coding: utf-8import contextlib@contextlib.contextmanager # define this function to support context with statement def myopen (filename, mode): f = open (filename, mode) try: yield f. readlines () # f. readlines () except t Exception as e: print e finally: f. close () # Return the f. close () to close the file if _ name _ = '_ main _': with myopen (r 'C: \ ip1.txt ', 'R') as f: for line in f: print line