python並發編程之進程池，線程池concurrent.futures

標籤：spool   帶來   nbsp   app   pid   去掉   stat   def   pen   

進程池與線程池

在剛開始學多進程或多線程時，我們迫不及待地基於多進程或多線程實現並發的通訊端通訊，然而這種實現方式的致命缺陷是：服務的開啟的進程數或線程數都會隨著並發的用戶端數目地增多而增多，

這會對服務端主機帶來巨大的壓力，甚至於不堪重負而癱瘓，於是我們必須對服務端開啟的進程數或線程數加以控制，讓機器在一個自己可以承受的範圍內運行，這就是進程池或線程池的用途，

例如進程池，就是用來存放進程的池子，本質還是基於多進程，只不過是對開啟進程的數目加上了限制

Python--concurrent.futures

1.concurent.future模組是用來建立並行的任務，提供了更進階別的介面，
為了非同步執行調用
2.concurent.future這個模組用起來非常方便，它的介面也封裝的非常簡單
3.concurent.future模組既可以實現進程池，也可以實現線程池
4.模組匯入進程池和線程池
from  concurrent.futures import ProcessPoolExecutor,ThreadPoolExecutor
p = ProcessPoolExecutor(max_works)對於進程池如果不寫max_works：預設的是cpu的數目
p = ThreadPoolExecutor(max_works)對於線程池如果不寫max_works：預設的是cpu的數目*5

基本方法

1、submit(fn, *args, **kwargs)非同步提交任務2、map(func, *iterables, timeout=None, chunksize=1) 取代for迴圈submit的操作3、shutdown(wait=True) 相當於進程池的pool.close()+pool.join()操作wait=True，等待池內所有任務執行完畢回收完資源後才繼續wait=False，立即返回，並不會等待池內的任務執行完畢但不管wait參數為何值，整個程式都會等到所有任務執行完畢submit和map必須在shutdown之前4、result(timeout=None)取得結果5、add_done_callback(fn)回呼函數

　

 進程池

from concurrent.futures import ProcessPoolExecutor,ThreadPoolExecutorfrom threading import currentThreadimport os,time,randomdef task(n):    print("%s is running " % os.getpid())    time.sleep(random.randint(1,3))    return n*2if __name__ == ‘__main__‘:    start = time.time()    executor = ProcessPoolExecutor(4)    res = []    for i in range(10):  # 開啟10個任務        future = executor.submit(task,i)  # 非同步提交任務        res.append(future)    executor.shutdown()  # 等待所有進程執行完畢    print("++++>")    for r in res:        print(r.result())  # 列印結果    end = time.time()    print(end - start)---------------------輸出2464 is running 9356 is running 10780 is running 9180 is running 2464 is running 10780 is running 9180 is running 9356 is running 10780 is running 9180 is running ++++>0246810121416186.643380165100098

線程池

from concurrent.futures import ProcessPoolExecutor,ThreadPoolExecutorfrom threading import currentThreadimport os,time,randomdef task(n):    print("%s is running " % currentThread().getName())    time.sleep(random.randint(1,3))    return n*2if __name__ == ‘__main__‘:    start = time.time()    executor = ThreadPoolExecutor(4)  # 線程池    res = []    for i in range(10):  # 開啟10個任務        future = executor.submit(task,i)  # 非同步提交任務        res.append(future)    executor.shutdown()  # 等待所有線程執行完畢    print("++++>")    for r in res:        print(r.result())  # 列印結果    end = time.time()    print(end - start)------------輸出<concurrent.futures.thread.ThreadPoolExecutor object at 0x00000000025B0DA0>_0 is running <concurrent.futures.thread.ThreadPoolExecutor object at 0x00000000025B0DA0>_1 is running <concurrent.futures.thread.ThreadPoolExecutor object at 0x00000000025B0DA0>_2 is running <concurrent.futures.thread.ThreadPoolExecutor object at 0x00000000025B0DA0>_3 is running <concurrent.futures.thread.ThreadPoolExecutor object at 0x00000000025B0DA0>_3 is running <concurrent.futures.thread.ThreadPoolExecutor object at 0x00000000025B0DA0>_1 is running <concurrent.futures.thread.ThreadPoolExecutor object at 0x00000000025B0DA0>_0 is running <concurrent.futures.thread.ThreadPoolExecutor object at 0x00000000025B0DA0>_2 is running <concurrent.futures.thread.ThreadPoolExecutor object at 0x00000000025B0DA0>_3 is running <concurrent.futures.thread.ThreadPoolExecutor object at 0x00000000025B0DA0>_1 is running ++++>0246810121416185.002286195755005

回呼函數

import requestsimport timefrom concurrent.futures import ThreadPoolExecutordef get(url):    print(‘GET {}‘.format(url))    response = requests.get(url)    time.sleep(2)    if response.status_code == 200:  # 200代表狀態：下載成功了        return {‘url‘: url, ‘content‘: response.text}def parse(res):    print(‘%s parse res is %s‘ % (res[‘url‘], len(res[‘content‘])))    return ‘%s parse res is %s‘ % (res[‘url‘], len(res[‘content‘]))def save(res):    print(‘save‘, res)def task(res):    res = res.result()    par_res = parse(res)    save(par_res)if __name__ == ‘__main__‘:    urls = [            ‘http://www.cnblogs.com/linhaifeng‘,            ‘https://www.python.org‘,            ‘https://www.openstack.org‘,        ]    pool = ThreadPoolExecutor(2)    for i in urls:        pool.submit(get, i).add_done_callback(task)#這裡的回呼函數拿到的是一個對象。得        #  先把返回的res得到一個結果。即在前面加上一個res.result() #誰好了誰去掉回呼函數                                # 回呼函數也是一種編程思想。不僅開線程池用，開線程池也用    pool.shutdown()  #相當於進程池裡的close和join-------------輸出GET http://www.cnblogs.com/linhaifengGET https://www.python.orghttp://www.cnblogs.com/linhaifeng parse res is 17426save http://www.cnblogs.com/linhaifeng parse res is 17426GET https://www.openstack.orghttps://www.python.org parse res is 48809save https://www.python.org parse res is 48809https://www.openstack.org parse res is 60632save https://www.openstack.org parse res is 60632

 

map

import requestsimport timefrom concurrent.futures import ThreadPoolExecutordef get(url):    print(‘GET {}‘.format(url))    response = requests.get(url)    time.sleep(2)    if response.status_code == 200:  # 200代表狀態：下載成功了        return {‘url‘: url, ‘content_len‘: len(response.text)}if __name__ == ‘__main__‘:    urls = [            ‘http://www.cnblogs.com/linhaifeng‘,            ‘https://www.python.org‘,            ‘https://www.openstack.org‘,        ]    pool = ThreadPoolExecutor(2)    res = pool.map(get, urls) #map取代了for+submit    pool.shutdown()   # 相當於進程池裡的close和join    print(‘=‘ * 30)    for r in res: # 返回的是一個迭代器        print(r)GET http://www.cnblogs.com/linhaifengGET https://www.python.orgGET https://www.openstack.org{‘url‘: ‘http://www.cnblogs.com/linhaifeng‘, ‘content_len‘: 17426}{‘url‘: ‘https://www.python.org‘, ‘content_len‘: 48809}{‘url‘: ‘https://www.openstack.org‘, ‘content_len‘: 60632}

　　

　　

 

python並發編程之進程池，線程池concurrent.futures