Python Learning _day35_ Concurrent Programming Multi-Process 3

One, the producer consumption model complements

Summarize:

---two roles in the producer Consumer model program: ① is responsible for production data (producer); ② is responsible for processing data (consumer)

---the role of producer consumer models: balancing the speed difference between producers and consumers.

---implementation: Producer--and consumer

As in the previous blog content about production consumption model content, after the process of producer production data, even if the consumer has fully obtained the data, the consumer program can not end, the main process or the producer after the end of the production process sent to the end of the customer password, the consumer program will end. But if there are multiple consumers and multiple producers, how can this be solved? There are two ways to do this:

1, according to the number of consumers to transmit the end signal (low)

 fromMultiprocessingImportProcess,queueImport TimeImportRandomImportOSdefproducer (name,q): forIinchRange (10): Res='%s%s'%(name,i) time.sleep (Random.randint (1, 3) ) Q.put (res)Print('%s was produced %s'%(Os.getpid (), res))defConsumer (name,q): whileTrue:res=Q.get ()if  notRes: Break        Print('%s ate%s'%(name,res))if __name__=='__main__': Q=Queue () P1=process (target=producer,args= ('steamed Bun', Q)) P2=process (target=producer,args= ('swill', Q)) P3=process (Target=producer, args= ('Bones', Q)) C1=process (target=consumer,args= ('Alex', Q)) C2=process (target=consumer,args= ('Egon', Q)) _p=[P1,P2,P3,C1,C2] forPinch_p:p.start () p1.join () P2.join () P3.join ( )" "ensure the end of the production process, send the end signal, the number of sending and the number of consumers consistent" "Q.put (None) Q.put (none)

2. Joinablequeue queue mechanism

Joinablequeue are basically similar to the queue queues, but the former queues allow the user of the project to notify the creator that the project has been successfully processed. The notification process is implemented using shared signals and condition variables. The object of a queue instance has the same method as Joinablequeue, which has the following methods in addition to this joinablequeue:

①q.join (): The producer calls this method to block until all items in the queue are processed. Blocking will persist until each item in the queue calls the Q.task_done () method

②q.task_done (): The consumer uses this method to signal that the returned item of Q.get () has been processed. If the number of times this method is called is greater than the number of items removed from the queue, a ValueError exception is thrown

 fromMultiprocessingImportProcess,joinablequeueImport TimeImportRandomdefproducer (name,food,q): forIinchRange (10): Res='%s%s'%(food,i) time.sleep (Random.randint (1, 3) ) Q.put (res)Print('%s was produced %s'%(Name,res)) Q.join ()#block the producer process to ensure that the consumer process finishes processing the data it produces at the end of the processdefConsumer (name,q): whileTrue:res=Q.get ()if  notRes: Break        Print('%s ate%s'%(Name,res)) Q.task_done ()if __name__=='__main__': Q=joinablequeue () P1=process (target=producer,args= (1,'steamed Bun', Q)) P2=process (target=producer,args= (2,'swill', Q)) P3= Process (Target=producer, args= (3,'Bones', Q)) C1=process (target=consumer,args= ('Alex', Q)) C2=process (target=consumer,args= ('Egon', Q)) " "The daemon ensures that the daemon ends at the end of the master process." "C1.daemon=True C2.daemon=True _p=[P1,P2,P3,C1,C2] forPinch_p:p.start () p1.join () P2.join () P3.join ( )

Second, shared memory (Manager)

Looking ahead, concurrent programming based on messaging is a trend, and even with threading, the recommended approach is to design a large collection of independent threads to exchange data through Message Queuing. This greatly reduces the need to use locking and other synchronization methods, and can be extended to distributed systems where inter- process communication should avoid the use of shared data as described in this section.

 fromMultiprocessingImportProcess,lock,managerdefWork (D,lock):" "locking is to ensure that data is modified one by one, to avoid multiple processes at the same time to get a starting data, resulting in data confusion" "With lock:d['Count']-=1if __name__=='__main__': Lock=Lock () m=Manager () d=m.dict ({'Count': 100})#Other data types can also be sexually shared, such as the list_p=[]     forIinchRange (10): P=process (target=work,args=(D,lock)) P.start () _p.append (p) forPinch_p:p.join ()Print(d)#{' count ':

Third, process pools (pool)

Multi-process is one of the main means to achieve concurrency, but usually there are the following problems: A. It is clear that the tasks that need to be performed concurrently are usually much larger than the number of cores; C. The process is too open, and efficiency decreases (the process of opening up requires system resources, and the processes that open the number of extra cores cannot be parallel).

Pool can provide a specified number of processes for the user to invoke, and when a new request is submitted to the pool, a new process is created to execute the request if it is not full, but if the number of processes in the pool has reached the specified maximum, the request waits until the process ends in the pool. Processes in the process pool are reused. The number of processes that perform the task is always not the one specified in the process pool.

1. Synchronous call (Apply)

Synchronous invocation: After the task is submitted, wait for the task to end in place, once the end can get the result immediately.

 fromMultiprocessingImportPoolImportTime,osImportRandomdefWork (n):Print('%s is working'%(Os.getpid ())) Time.sleep (Random.randint (1,3))    returnN**2if __name__=='__main__': P=pool (4)                                   #从无到有开启4个进程, and it's been 4 of these processes

 for  in range:
    Res=p.apply (work,args= (i))                 # Wait until the process finishes executing and can get the result before the next process is opened, equivalent to the serial
    Print(res)

2. Asynchronous Invocation (Apply_async)

　　Asynchronous invocation: After the task is submitted, it will not wait for the task to end in place, and will continue to submit the next task until all tasks are finished before the get result is obtained.

 fromMultiprocessingImportPoolImportTime,osImportRandomdefWork (n):Print('%s is working'%(Os.getpid ())) Time.sleep (Random.randint (1,3))    returnN**2if __name__=='__main__': P=pool (4)#4 processes have been opened from scratch, and these 4 processes have beenresult=[]     forIinchRange (10): obj=p.apply_async (work,args= (i,))#submit a task, not in situ etc resultsresult.append (obj)" "Asynchronous Apply_async Usage: If you use asynchronous-committed tasks, the main process needs to use Jion, wait for the process pool tasks to finish processing, and then collect the results with get, otherwise, the main process ends, the process pool may not be able to execute, and then it ends together." "p.close () p.join ( ) forObjinchResult:Print(Obj.get ())" "use get to get the result of apply_aync, if apply, there is no get method, because apply is synchronous execution, get the result immediately, do not need get at all" "

Four, callback function

　　The process pool executes a process that obtains data and immediately requests that the main process be notified to parse the data. The main process calls a function to process, and this function is called a callback function, which requires the result of the process pool process to be the parameters of the callback function.

Crawler Examples:

 fromMultiprocessingImportpool,processImportRequestsImportOSImportTime,randomdefget (URL):Print('%s GET%s'%(Os.getpid (), URL)) Response=requests.get (URL) time.sleep (Random.randint (1,3))    ifResponse.status_code = = 200:        Print('%s Done%s'%(Os.getpid (), URL))return{'URL': URL,'text': Response.text}defParse (DIC):Print('%s PARSE%s'% (Os.getpid (), dic['URL'])) Time.sleep (1) Res='%s:%s\n'% (dic['URL'],len (dic['text'])) with open ('Db.txt','a') as F:f.write (res)if __name__=='__main__': URLs=[        'https://www.baidu.com',        'https://www.python.org',        'https://www.openstack.org',        'https://help.github.com/',        'http://www.sina.com.cn/'] P=pool (2)     forUrlinchUrls:p.apply_async (Get,args= (URL,), callback=parse)#The main process is responsible for the work of the callback functionp.close () p.join ( )

We can put the time-consuming (blocking) task into the process pool and then specify the callback function (the main process is responsible for executing) so that the main process eliminates the I/O process when executing the callback function, and the result of the task is directly obtained. If you wait for all the tasks in the process pool to finish executing in the main process and then process the results uniformly, you do not need a callback function.

Python Learning _day35_ Concurrent Programming Multi-Process 3