Python process pool and thread pool

Code

When writing a crawler, performance is consumed primarily in IO requests, which inevitably cause a wait when the URL is requested in single-threaded mode, which slows down the overall request.

ImportRequests fromConcurrent.futuresImportThreadpoolexecutor#Introducing thread pool modulesdefasyns_url (URL):Try: Response=requests.get (URL)exceptException as E:Print('Exception Results', Response.url,response.content)Print('Get Results', Response.url, response.content) url_list={    'http://www.baidu.com',    'http://www/google.com',    'http://dig.chouti.com',    'http://www.bing.com'}pool=threadpoolexecutor (5) forUrlinchurl_list:Print('Start Request', URL) pool.submit (asyns_url,url) pool.shutdown (Wait=true)#terminating a thread

 fromConcurrent.futuresImportProcesspoolexecutorImportRequestsdeffetch_async (URL): Response=requests.get (URL)returnresponseurl_list= ['http://www.github.com','http://www.bing.com']pool= Processpoolexecutor (5) forUrlinchurl_list:pool.submit (fetch_async, URL) pool.shutdown (wait=True)3. Multi-Process Execution

Multi-Process

 fromConcurrent.futuresImportProcesspoolexecutorImportRequestsdeffetch_async (URL): Response=requests.get (URL)returnResponsedefCallback (future):Print(Future.result ()) Url_list= ['http://www.github.com','http://www.bing.com']pool= Processpoolexecutor (5) forUrlinchurl_list:v=pool.submit (fetch_async, URL) v.add_done_callback (callback) Pool.shutdown (Wait=True)3. Multi-process + callback function execution

Multi-process + callback function

Python process pool and thread pool