Thread Pool ********Python standard module--Concurrent.futures1The introduction of the Concurrent.futures module provides a highly encapsulated asynchronous calling interface Threadpoolexecutor: The thread pool, which provides an asynchronous call to Processpoolexecutor: The process pool, which provides asynchronous calls both Implement the same interface, which isDefined by the abstract Executorclass.2. Basic Methods#Submit (FN, *args, **kwargs)Asynchronous Submit Task#Map (func, *iterables, Timeout=none, chunksize=1)action to replace the for loop submit#shutdown (wait=true)Equivalent to Process pool pool.close () +Pool.join () operation wait=True to wait for all tasks in the pool to finish after the resource has been reclaimed before the wait is resumed=False, returns immediately without waiting for the task in the pool to complete but regardless of the wait parameter value, the entire program waits until all tasks have been completed submit and map must precede shutdown#result (Timeout=none)Get Results#add_done_callback (FN)callback functionprocesspoolexecutor****#Introduction" "The Processpoolexecutor class is an Executor subclass, uses a pool of processes to execute calls asynchronously. P Rocesspoolexecutor uses the multiprocessing module, which allows it to side-step the Global interpreter Lock but also mean s that is picklable objects can be executed and Returned.class Concurrent.futures.ProcessPoolExecutor (max_workers=none , Mp_context=none) An Executor subclass that executes calls asynchronously using a pool of in most max_workers processes. If Max_workers is None or not given, it'll default to the number of processors on the machine. If max_workers is lower or equal to 0,then a valueerror would be raised." "#usage fromConcurrent.futuresImportThreadpoolexecutor, ProcesspoolexecutorImportOS, time, Randomdeftask (n):Print('%s is runing'%os.getpid ()) Time.sleep (Random.randint (1, 3)) returnn * * 2if __name__=='__main__': Executor= Processpoolexecutor (Os.cpu_count () + 1) Futures= [] forIinchRange (11): Future=Executor.submit (Task, I) futures.append (future) Executor.shutdown (True)Print('+++>') forFutureinchFutures:Print(Future.result ())threadpoolexecutor****#Introduction" "Threadpoolexecutor is an Executor subclass, uses a pool of threads to execute calls Asynchronously.class Concurren T.futures.threadpoolexecutor (Max_workers=none, thread_name_prefix= ") an Executor subclass that uses a pool of in most Max _workers threads to execute calls asynchronously. Changed in version 3.5:if max_workers are None or not given, it'll default to the number of processors on the machine, M Ultiplied by 5, assuming that Threadpoolexecutor was often used to overlap I/o instead of CPU work and the number of worker s should be higher than the number of workers for processpoolexecutor.new in version 3.6:the thread_name_prefix argument Was added to allow users to control the threading. Thread names for worker threads created by the pool for easier debugging." "#usage#Same as Processpoolexecutormap**** fromConcurrent.futuresImportThreadpoolexecutor, ProcesspoolexecutorImportOS, time, Randomdeftask (n):Print('%s is runing'%os.getpid ()) Time.sleep (Random.randint (1, 3)) returnn * * 2if __name__=='__main__': Executor= Threadpoolexecutor (Os.cpu_count () * 5) #For i in range: #future=executor.submit (task,i)Executor.map (Task, range (1, 42))#Map replaces the For+submitcallback function * * * * fromConcurrent.futuresImportThreadpoolexecutor, Processpoolexecutor fromMultiprocessingImportPoolImportRequestsImportJSONImportOSdefget_page (URL):Print('< process%s> Get%s'%(Os.getpid (), URL)) Respone=requests.get (URL)ifRespone.status_code = = 200: return{'URL': URL,'text': Respone.text}defParse_page (RES): Res=Res.result ()Print('< process%s> Parse%s'% (Os.getpid (), res['URL'])) Parse_res='url:<%s> size:[%s]\n'% (res['URL'], Len (res['text'])) with open ('Db.txt','a') as F:f.write (parse_res)if __name__=='__main__': URLs= [ 'https://www.baidu.com', 'https://www.python.org', 'https://www.openstack.org', 'https://help.github.com/', 'http://www.sina.com.cn/' ] #P=pool (3) #For URL in URLs: #p.apply_async (get_page,args= (URL,), callback=pasrse_page) #p.close () #P.join ()P= Processpoolexecutor (3) forUrlinchurls:p.submit (get_page, URL). Add_done_callback (Parse_page)#Parse_page got a future object. obj, need to get results with Obj.result ()
python-thread Pool