Concurrent programming Multi-Process 3 (producer and consumer model) callback function

One, the producer consumption model complements

Summarize:

---two roles in the producer Consumer model program: ① is responsible for production data (producer); ② is responsible for processing data (consumer)

---the role of producer consumer models: balancing the speed difference between producers and consumers.

---implementation: Producer--and consumer

As in the previous blog content about production consumption model content, after the process of producer production data, even if the consumer has fully obtained the data, the consumer program can not end, the main process or the producer after the end of the production process sent to the end of the customer password, the consumer program will end. But if there are multiple consumers and multiple producers, how can this be solved? There are two ways to do this:

1, according to the number of consumers to transmit the end signal (low)

 fromMultiprocessingImportProcess,queueImportTime,random,osdefProcducer (q): forIinchRange (10): Res='Bun%s'%I time.sleep (0.5) Q.put (res)Print('%s was produced %s'%(Os.getpid (), res))defConsumer (q): whileTrue:res=Q.get ()ifRes isNone: Break        Print('%s Eat%s'%( os.getpid (), res)) Time.sleep (Random.randint (2,3))if __name__=='__main__': Q=Queue () p=process (target=procducer,args=(q,)) C=process (target=consumer,args=(q,)) P.start () C.start () P.join () q.put (None)Print('Master')

 fromMultiprocessingImportProcess,queueImport TimeImportRandomImportOSdefproducer (name,q): forIinchRange (10): Res='%s%s'%(name,i) time.sleep (Random.randint (1, 3) ) Q.put (res)Print('%s was produced %s'%(Os.getpid (), res))defConsumer (name,q): whileTrue:res=Q.get ()if  notRes: Break        Print('%s ate%s'%(name,res))if __name__=='__main__': Q=Queue () P1=process (target=producer,args= ('Chocolate', Q)) P2=process (target=producer,args= ('Donuts', Q)) P3=process (Target=producer, args= ('Cream Cake', Q)) C1=process (target=consumer,args= ('Alex', Q)) C2=process (target=consumer,args= ('Egon', Q)) _p=[P1,P2,P3,C1,C2] forPinch_p:p.start () p1.join () P2.join () P3.join ( )" "ensure the end of the production process, send the end signal, the number of sending and the number of consumers consistent" "Q.put (None) Q.put (none)

Oh, my God.

2. Joinablequeue queue mechanism

Joinablequeue are basically similar to the queue queues, but the former queues allow the user of the project to notify the creator that the project has been successfully processed. The notification process is implemented using shared signals and condition variables. The object of a queue instance has the same method as Joinablequeue, which has the following methods in addition to this joinablequeue:

①q.join (): The producer calls this method to block until all items in the queue are processed. Blocking will persist until each item in the queue calls the Q.task_done () method

②q.task_done (): The consumer uses this method to signal that the returned item of Q.get () has been processed. If the number of times this method is called is greater than the number of items removed from the queue, a ValueError exception is thrown

 fromMultiprocessingImportProcess,joinablequeueImport TimeImportRandomdefproducer (name,food,q): forIinchRange (10): Res='%s%s'%(food,i) time.sleep (Random.randint (1, 3) ) Q.put (res)Print('%s was produced %s'%(Name,res)) Q.join ()#block the producer process to ensure that the consumer process finishes processing the data it produces at the end of the processdefConsumer (name,q): whileTrue:res=Q.get ()if  notRes: Break        Print('%s ate%s'%(Name,res)) Q.task_done ()if __name__=='__main__': Q=joinablequeue () P1=process (target=producer,args= (1,'Chocolate', Q)) P2=process (target=producer,args= (2,'Cream Cake', Q)) P3= Process (Target=producer, args= (3,'Gourd', Q)) C1=process (target=consumer,args= ('Lishi', Q)) C2=process (target=consumer,args= ('Jassin', Q)) " "The daemon ensures that the daemon ends at the end of the master process." "C1.daemon=True C2.daemon=True _p=[P1,P2,P3,C1,C2] forPinch_p:p.start () p1.join () P2.join () P3.join ( )

Second, callback function

　　The process pool executes a process that obtains data and immediately requests that the main process be notified to parse the data. The main process calls a function to process, and this function is called a callback function, which requires the result of the process pool process to be the parameters of the callback function.

Crawler Examples:

Thread pool

ImportRequests fromConcurrent.futuresImportthreadpoolexecutor (thread pool), Processpoolexecutor (process pool) fromThreadingImportCurrent_threadImport TimeImportOSdefGet (URL):#Download    Print('%s GET%s'%(Current_thread (). GetName (), URL)) Response=requests.get (URL) time.sleep (3)    ifResponse.status_code = = 200:#fixed, =200 means download complete        return{'URL': URL,'text': Response.text}defParse (obj):#parsingres=Obj.result ()Print('[%s] <%s> (%s)'% (Current_thread (). GetName (), res['URL'],len (res['text']))) if __name__=='__main__': URLs= [        'https://www.python.org',        'https://www.baidu.com',        'https://www.jd.com',        'https://www.tmall.com',] t=threadpoolexecutor (2)     forUrlinchUrls:t.submit (Get,url). Add_done_callback (parse) t.shutdown (wait=True)Print('Master', Os.getpid ())

We can put the time-consuming (blocking) task into the process pool and then specify the callback function (the main process is responsible for executing) so that the main process eliminates the I/O process when executing the callback function, and the result of the task is directly obtained. If you wait for all the tasks in the process pool to finish executing in the main process and then process the results uniformly, you do not need a callback function.

Process Pool

ImportRequests fromConcurrent.futuresImportThreadpoolexecutor,processpoolexecutorImport TimeImportOSdefget (URL):Print('%s GET%s'%(Os.getpid (), URL)) Response=requests.get (URL) time.sleep (3)    ifResponse.status_code = = 200:        return{'URL': URL,'text': Response.text}defParse (obj): res=Obj.result ()Print('[%s] <%s> (%s)'% (Os.getpid (), res['URL'],len (res['text'])))if __name__=='__main__': URLs= [        'https://www.python.org',        'https://www.baidu.com',        'https://www.jd.com',        'https://www.tmall.com',] t=processpoolexecutor (2)     forUrlinchUrls:t.submit (Get,url). Add_done_callback (parse) t.shutdown (wait=True)Print('Master', Os.getpid ())

Concurrent programming Multi-Process 3 (producer and consumer model) callback function