42 _ concurrent programming-jionablequeue

I. producer and consumer model in the thread world, the producer is the thread that produces data, and the consumer is the thread that consumes data. In multi-threaded development, if the producer processing speed is very fast and the Consumer processing speed is very slow, the producer must wait until the consumer completes processing before continuing to produce data. Similarly, if the processing capability of a consumer is greater than that of a producer, the consumer must wait for the producer. To solve this problem, the producer and consumer models were introduced. The producer consumer model uses a container to solve the strong coupling problem between producers and consumers. Producers and consumers do not directly communicate with each other, but communicate by blocking the queue. Therefore, after the producer completes data production, the producer does not have to wait for the consumer to process the data and directly throws it to the blocking queue. the consumer does not seek data from the producer, instead, the block queue is directly retrieved from the blocking queue, which is equivalent to a buffer zone, balancing the processing capabilities of producers and consumers, in addition, I can balance the production speed and consumption speed to find out how many producers can provide sufficient services to the number of consumers, and then enable multi-process operations, these processes are all used to obtain or add data to a blocked queue or a buffer. Ii. producer consumer model implementation 1. Implement a producer consumer model based on queues

1 From multiprocessing import process, queue 2 import time, random, OS 3 def consumer (q): 4 While true: 5 res = Q. get () 6 time. sleep (random. randint (1, 3) 7 print ('\ 033 [45 m % s eat % s \ 033 [0m' % (OS. getpid (), Res) 8 9 def producer (q): 10 for I in range (10): 11 time. sleep (random. randint (1, 3) 12 res = 'steamed stuffed bun % s' % I13 Q. put (RES) 14 print ('\ 033 [44 m % s produced % s \ 033 [0m' % (OS. getpid (), Res) 15 16 if _ name _ = '_ main _': 17 q = Queue () 18 # Producers: that is, the chefs 19 p1 = process (target = producer, argS = (Q,) 20 21 # consumers: Food users 22 C1 = process (target = consumer, ARGs = (Q,) 23 24 # Start 25 p1.start () 26 c1.start () 27 print ('main ')

Summary:

1 # Summary of producer and consumer models 2 # There are two types of roles in the Program 3 one is responsible for producing data (producers) 4 one is responsible for processing data (consumers) 5 6 # introduce the producer and consumer model to solve the problem: 7. Balance the working ability between producers and consumers, so as to improve the overall data processing speed of the program. 8 9 # How to achieve it: 10 producer <--> queue <--> consumer 11 # producer consumer model implements program decoupling and

Problem:

Through the above sample queue-based producer and consumer code, we found a problem: the main process will never end, because the producer P ends after production, however, after the consumer C has null Q, it is always in an endless loop and stuck in Q. get. The solution is to let the producer send an end signal to the queue after the production is complete, so that the consumer can break an endless loop after receiving the end signal.

2. The main process sends an end signal. Note: The end signal is none and does not have to be sent by the producer. The main process can also send an end signal, however, the main process must send this signal only after the producer ends.

1 From multiprocessing import process, queue 2 import time 3 def shenchan (q): 4 for I in range (10): # produce a steamed stuffed bun 5 time every 1 second. sleep (1) 6 print ('production % s Steamed Stuffed Bun '% I) 7 Q. put (I) # produce a put Queue (buffer) 8 9 10 def Xiaofei (q): 11 while time. sleep (0.5) # Get the steamed stuffed bun from the queue all the time, 13 if Q every 0.5 seconds. get () = none: # if none is obtained from the queue, the system will not exit. The none signal is sent by the main process to 14 break15 else: 16 print ('consumer eat % s Steamed Stuffed Bun '% Q. get () # If the consumer gets faster than the producer's steamed stuffed bun, The get, wait for the consumer to put the Steamed Stuffed Bun into the queue and then take 17 18 if _ name _ = '_ main _': 19 20 q = Queue (10) 21 SC = process (target = shenchan, argS = (Q,) 22 SC. start () 23 XF = process (target = Xiaofei, argS = (Q,) 24 XF. start () 25 SC. join () # Set join to block it. Let the producer and consumer process finish executing 26 Q. put (none) # Try to queue China none

3. Implement joinablequeue ([maxsize]) through joinablequeue ([maxsize]): this is like a queue object, but the queue allows the project user to notify the builder that the project has been successfully processed. The notification process is implemented using shared signals and conditional variables. Parameter description: maxsize is the maximum number of items allowed in the queue. If it is omitted, there is no size limit. Method Description: In addition to the same method as the queue object, the instance P of joinablequeue also has: Q. task_done (): the user uses this method to send a signal, indicating Q. the returned project of get () has been processed. If the number of times this method is called is greater than the number of items deleted from the queue, A valueerror exception Q. join (): The producer calls this method for blocking until all items in the queue are processed. Blocking continues until every project in the queue calls the Q. task_done () method, that is, all the data in the queue is taken by get.

1 # producer consumer model 2 import time 3 from multiprocessing import process, queue, joinablequeue 4 5 def producer (q): 6 for I in range (1, 11): 7 time. sleep (0.5) 8 print ('produced Steamed Stuffed Bun % s' % I) 9 q. put (I) 10 Q. join () 11 print ('Wait here ') 12 def consumer (q): 13 while time. sleep (1) 15 S = Q. get () 16 print ('consumer has eaten % s Steamed Stuffed Bun '% s) 17 Q. task_done () # Send a task end signal to the Q object 18 19 if _ name _ = '_ main _': 20 # simulate the buffer through the queue, size: 2021 q = joinablequeue (20) 22 # producer process 23 pro_p = process (target = producer, argS = (Q,) 24 pro_p.start () 25 # consumer process 26 con_p = process (target = consumer, argS = (Q,) 27 con_p.daemon = true #28 con_p.start () 29 pro_p.join () 30 print ('master process termination ')

 

# Import timefrom multiprocessing import process, queue, joinablequeue def producer (q): for I in range (): time. sleep (0.5) print ('generated Steamed Stuffed Bun % s' % I) Q. put (I) Q. join () print ('Wait here ') def consumer (q): While 1: time. sleep (1) S = Q. get () print ('consumer has eaten % s Steamed Stuffed Bun '% s) Q. task_done () # Send a task end signal to the Q object if _ name _ = '_ main _': # simulate the buffer through the queue, set the size to 20 q = joinablequeue (20) # producer process pro_p = process (target = producer, argS = (Q,) pro_p.start () # consumer process con_p = process (target = consumer, argS = (Q,) con_p.daemon = true # con_p.start () pro_p.join () print ('master process termination ')

 

42 _ concurrent programming-jionablequeue