Python自訂進程池執行個體分析【生產者、消費者模型問題】_python

本文執行個體分析了Python自訂進程池。分享給大家供大家參考，具體如下：

代碼說明一切：

#encoding=utf-8#author: walker#date: 2014-05-21#function: 自訂進程池遍曆目錄下檔案from multiprocessing import Process, Queue, Lockimport time, os#消費者class Consumer(Process):  def __init__(self, queue, ioLock):    super(Consumer, self).__init__()    self.queue = queue    self.ioLock = ioLock  def run(self):    while True:      task = self.queue.get()  #隊列中無任務時，會阻塞進程      if isinstance(task, str) and task == 'quit':        break;      time.sleep(1)  #假定任務處理需要1秒鐘      self.ioLock.acquire()      print( str(os.getpid()) + ' ' + task)      self.ioLock.release()    self.ioLock.acquire()    print 'Bye-bye'    self.ioLock.release()#生產者def Producer():  queue = Queue()  #這個隊列是進程/安全執行緒的  ioLock = Lock()  subNum = 4  #子進程數量  workers = build_worker_pool(queue, ioLock, subNum)  start_time = time.time()  for parent, dirnames, filenames in os.walk(r'D:\test'):    for filename in filenames:      queue.put(filename)      ioLock.acquire()      print('qsize:' + str(queue.qsize()))      ioLock.release()      while queue.qsize() > subNum * 10: #控制隊列中任務數量        time.sleep(1)  for worker in workers:    queue.put('quit')  for worker in workers:    worker.join()  ioLock.acquire()  print('Done! Time taken: {}'.format(time.time() - start_time))  ioLock.release()#建立進程池def build_worker_pool(queue, ioLock, size):  workers = []  for _ in range(size):    worker = Consumer(queue, ioLock)    worker.start()    workers.append(worker)  return workersif __name__ == '__main__':  Producer()

ps：

self.ioLock.acquire()...self.ioLock.release()

可用：

with self.ioLock:  ...

替代。

再來一個好玩的例子：

#encoding=utf-8#author: walker#date: 2016-01-06#function: 一個多進程的好玩例子import os, sys, timefrom multiprocessing import Poolcur_dir_fullpath = os.path.dirname(os.path.abspath(__file__))g_List = ['a']#修改全域變數g_Listdef ModifyDict_1():  global g_List  g_List.append('b')#修改全域變數g_Listdef ModifyDict_2():  global g_List  g_List.append('c')#處理一個def ProcOne(num):  print('ProcOne ' + str(num) + ', g_List:' + repr(g_List))#處理所有def ProcAll():  pool = Pool(processes = 4)  for i in range(1, 20):    #ProcOne(i)    #pool.apply(ProcOne, (i,))    pool.apply_async(ProcOne, (i,))  pool.close()  pool.join()ModifyDict_1() #修改全域變數g_Listif __name__ == '__main__':  ModifyDict_2() #修改全域變數g_List  print('In main g_List :' + repr(g_List))  ProcAll()

Windows7 下啟動並執行結果：

λ python3 demo.pyIn main g_List :['a', 'b', 'c']ProcOne 1, g_List:['a', 'b']ProcOne 2, g_List:['a', 'b']ProcOne 3, g_List:['a', 'b']ProcOne 4, g_List:['a', 'b']ProcOne 5, g_List:['a', 'b']ProcOne 6, g_List:['a', 'b']ProcOne 7, g_List:['a', 'b']ProcOne 8, g_List:['a', 'b']ProcOne 9, g_List:['a', 'b']ProcOne 10, g_List:['a', 'b']ProcOne 11, g_List:['a', 'b']ProcOne 12, g_List:['a', 'b']ProcOne 13, g_List:['a', 'b']ProcOne 14, g_List:['a', 'b']ProcOne 15, g_List:['a', 'b']ProcOne 16, g_List:['a', 'b']ProcOne 17, g_List:['a', 'b']ProcOne 18, g_List:['a', 'b']ProcOne 19, g_List:['a', 'b']

Ubuntu 14.04下啟動並執行結果：

In main g_List :['a', 'b', 'c']ProcOne 1, g_List:['a', 'b', 'c']ProcOne 2, g_List:['a', 'b', 'c']ProcOne 3, g_List:['a', 'b', 'c']ProcOne 5, g_List:['a', 'b', 'c']ProcOne 4, g_List:['a', 'b', 'c']ProcOne 8, g_List:['a', 'b', 'c']ProcOne 9, g_List:['a', 'b', 'c']ProcOne 7, g_List:['a', 'b', 'c']ProcOne 11, g_List:['a', 'b', 'c']ProcOne 6, g_List:['a', 'b', 'c']ProcOne 12, g_List:['a', 'b', 'c']ProcOne 13, g_List:['a', 'b', 'c']ProcOne 10, g_List:['a', 'b', 'c']ProcOne 14, g_List:['a', 'b', 'c']ProcOne 15, g_List:['a', 'b', 'c']ProcOne 16, g_List:['a', 'b', 'c']ProcOne 17, g_List:['a', 'b', 'c']ProcOne 18, g_List:['a', 'b', 'c']ProcOne 19, g_List:['a', 'b', 'c']

可以看見Windows7下第二次修改沒有成功，而Ubuntu下修改成功了。據uliweb作者limodou講，原因是Windows下是充重啟實現的子進程；Linux下是fork實現的。

更多關於Python相關內容感興趣的讀者可查看本站專題：《Python URL操作技巧總結》、《Python圖片操作技巧總結》、《Python資料結構與演算法教程》、《Python Socket編程技巧總結》、《Python函數提示總結》、《Python字串操作技巧匯總》、《Python入門與進階經典教程》及《Python檔案與目錄操作技巧匯總》

希望本文所述對大家Python程式設計有所協助。