What is the concept of thread pool? In object-oriented programming, it takes a lot of time to create and destroy objects, because creating an object requires obtaining memory resources or other resources. What is the concept of thread pool?
In object-oriented programming, it takes a lot of time to create and destroy objects, because creating an object requires obtaining memory resources or other resources. Even more so in Java, virtual machines will try to track every object so that garbage collection can be performed after the object is destroyed. Therefore, one way to improve service program efficiency is to minimize the number of times objects are created and destroyed, especially the creation and destruction of resource-consuming objects. How to use existing objects to serve is a key problem that needs to be solved. In fact, this is the reason why some "pooled resources" technologies are generated.
I understand that the thread pool is a unit that stores many threads and has a corresponding task queue. The entire execution process is actually to use a limited number of threads in the thread pool to finish the tasks in the task queue. The advantage of this is that you do not need to create a thread for each task, because when you create 100th threads to execute 100th tasks, maybe 50 threads have finished working. Therefore, we reuse threads to execute tasks and reduce the overhead of system resources.
An inappropriate analogy is that there are 100 hosts that need to be moved from the first floor to the second floor. you don't need to ask 100 people for help, you only need to call ten or twenty people. each person is allocated ten or five people, or even whoever moves quickly, who knows more about the unknown. (This is like .....)
In any case, we generally understand the concept of a thread pool. So how to implement it in python?
The code is as follows:
#! /Usr/bin/env python #-*-coding: UTF-8-*-# ref_blog: http://www.open-open.com/home/space-5679-do-blog-id-3247.htmlimport Queueimport threadingimport timeclass WorkManager (object): def _ init _ (self, work_num = 1000, thread_num = 2): self. work_queue = Queue. queue () self. threads = [] self. _ init_work_queue (work_num) self. _ init_thread_pool (thread_num) "initialization thread" def _ init_thread_pool (self, thread_num): for I in range (thread_num): self. threads. append (Work (self. work_queue) "initialize work queue" def _ init_work_queue (self, jobs_num): for I in range (jobs_num): self. add_job (do_job, I) "add a job to team" def add_job (self, func, * args): self. work_queue.put (func, list (args) # after a task is queued, the Queue implements the synchronization mechanism "check the remaining Queue tasks" def check_queue (self): return self. work_queue.qsize () "Waiting for all threads to run" def wait_allcomplete (self): for item in self. threads: if item. isAlive (): item. join () class Work (threading. thread): def _ init _ (self, work_queue): threading. thread. _ init _ (self) self. work_queue = work_queue self. start () def run (self): # endless loop, so that the created thread can be closed and exited under certain conditions. while True: try: do, args = self. work_queue.get (block = False) # The task is asynchronously queued. the Queue implements the synchronization mechanism do (args) self. work_queue.task_done () # notify the system task to complete failed t Exception, e: print str (e) break # the specific task def do_job (args): print args time. sleep (0.1) # simulate the processing time print threading. current_thread (), list (args) if _ name _ = '_ main _': start = time. time () work_manager = WorkManager (10, 2) # or work_manager = WorkManager (10000, 20) work_manager.wait_allcomplete () end = time. time () print "cost all time: % s" % (end-start)
This code is clear and easy to understand.
The entire code has only two classes: WorkManager and Work. The former is indeed named as shown in. it is a manager that manages thread pools and task queues, while the latter is a specific thread.
Its entire running logic is to allocate the specified task volume and number of threads to the WorkManager, and then each thread obtains the task from the task queue for execution until there is no task in the queue. The internal synchronization mechanism of Queue is also used here (so far the synchronization mechanism has not been studied yet ).
To sum up the role of such a thread pool, this will never last for my original purpose, because I need to control the thread startup and stop on the web page, this thread pool seems to be used only to send and complete tasks. However, I think it does not play a role in controlling the thread, but its concurrent task execution function is quite good. it may be used to crawl the web page.