Producer-consumer models in Python

Know the knowledge point: 1, Guardian process:

• What is a daemon process:

The daemon is actually a ' subprocess ', which is accompanied by a daemon that dies when the code of the main process finishes running.

• Why use Daemons:

When the code within the child process does not exist after the parent process code has finished running, the process should be set as the daemon and will die after the parent process code ends

From multiprocessing import Process

Import Time,os

def task (name):
Print ('%s is running '%name)
Time.sleep (3)

if __name__ = = ' __main__ ':
P1=process (target=task,args= (' Guardian process ',))
P2=process (target=task,args= (' normal sub-process ',))
P1.daemon=true # Be sure to put it before P.start ()
P1.start ()
P2.start ()
Print (' master ')

Examples of daemon processes

Here are some examples of how daemons can confuse people:

#主进程代码运行完毕, the daemon will be over.
From multiprocessing import Process
Import time
def foo ():
Print (123)
Time.sleep (1)
Print ("end123")

def bar ():
Print (456)
Time.sleep (3)
Print ("end456")

if __name__ = = ' __main__ ':
P1=process (Target=foo)
P2=process (Target=bar)

P1.daemon=true
P1.start ()
P2.start ()
Print ("main-------")

‘‘‘
Main-------
55W
enn456
‘‘‘

‘‘‘
Main-------
123
55W
enn456
‘‘‘

‘‘‘
123
Main-------
55W
end456
‘‘‘

2, Mutual exclusion Lock:

Mutex: You can change the part of the code that will perform the task (the code that only involves modifying the shared data) into a serial

Join: Is the overall serial of all code to perform the task

Emphasis: Must be lock.acquire () once, then Lock.release () released once to continue Lock.acquire (), cannot continuous lock.acquire (). No program stops in place.

The premise: The principle of both is the same, is to be parallel to the serial, so as to ensure orderly (in multiple programs to share a resource, in order to ensure orderly chaos, need to turn the concurrency into serial)

Difference One: Join is executed in the order specified by the person, and the mutex is so the process competes equally, who first grabs who executes

Difference Two: The mutex can let a part of the code (modify the shared data code) serial, and the join can only be the whole code serial (see the ticket system)

From multiprocessing import Process,lock
Import JSON
Import OS
Import time
Import Random

def check ():
Time.sleep (1) # Analog Network Latency
With open (' db.txt ', ' RT ', encoding= ' utf-8 ') as F:
Dic=json.load (f)
Print ('%s ' check see remaining votes [%s] '% (Os.getpid (), dic[' Count ']))

Def get ():
With open (' db.txt ', ' RT ', encoding= ' utf-8 ') as F:
Dic=json.load (f)
Time.sleep (2)
If dic[' count '] > 0:
# There are tickets
dic[' Count ']-=1
Time.sleep (Random.randint (1,3))
With open (' db.txt ', ' wt ', encoding= ' utf-8 ') as F:
Json.dump (DIC,F)
Print ('%s successfully purchased '%os.getpid ())
Else
Print ('%s No more tickets '%os.getpid ())

def task (Mutex):
# ticket
Check ()

#购票
Mutex.acquire () # Mutex cannot be acquire continuously, must be release to re-acquire
Get ()
Mutex.release ()

# with Mutex:
# get ()

if __name__ = = ' __main__ ':
Mutex=lock ()
For I in range (10):
P=process (target=task,args= (mutex,))
P.start ()
# P.join ()

Mock Grab Ticket

3. IPC Communication mechanism

Communication between processes must find a medium that must meet

1, is shared by all processes

2. Must be memory space

Additional: Help us to handle the lock problem automatically

A, from multiprocessing import Manager (shared memory, but to solve the problem of the lock yourself)

B. Queue sharing in IPC, memory, auto handle lock problem (most commonly used)

C, IPC Pipeline (pipe), sharing, memory, need to solve the problem of the lock itself

A, using the manager (to understand the knowledge points)

From multiprocessing import Process,manager,lock
Import time

Mutex=lock ()

def task (Dic,lock):
Lock.acquire ()
temp=dic[' num ']
Time.sleep (0.1)
dic[' num ']=temp-1
Lock.release ()

if __name__ = = ' __main__ ':
M=manager ()
Dic=m.dict ({' num ': 10})

L=[]
For I in range (10):
P=process (target=task,args= (Dic,mutex))
L.append (P)
P.start ()
For P in L:
P.join ()
Print (DIC)

B. Queue with queues

1) Shared Space

2) is the memory space

3) automatically help us to handle the locking problem.

From multiprocessing import Queue
Q=queue (3) #设置队列中maxsize个数为三
Q.put (' first ')
Q.put ({' Second ': None})
Q.put (' three ')
# Q.put (4) #阻塞. No error, the program card in the waiting queue to clear a value. Default Blok=true
Print (Q.get ())
Print (Q.get ())
Print (Q.get ())

Emphasize:
1, the queue used to save the process of communication between the message, the amount of data should not be too large
2. The memory limit of the maxsize value is meaningless

Know:
Q=queue (3)
Q.put (' first ', block=false)
Q.put (' second ', block=false)
Q.put (' third ', block=false)
Q.put (' fourth ', block=false) #报错 queue. Full

Q.put (' first ', block=true)
Q.put (' second ', block=true)
Q.put (' third ', block=true)
Q.put (' fourth ', block=true,timeout=3) #等待3秒后若还进不去报错. Note that timeout cannot be block=false with

Q.get (Block=false)
Q.get (Block=false)
Q.get (Block=false)
Q.get (block=false) #报错 queue. Empty

Q.get (Block=true)
Q.get (Block=true)
Q.get (Block=true)
Q.get (block=true,timeout=2) #等待2秒后还取不出东西则报错. Note that timeout cannot be block=false with

Know

4. Producer and Consumer models

The model contains two important types of roles:

1. Producer: The task of creating data is likened to producer

2, consumers: To receive the data produced by the producers to do further processing, such figures are likened to consumers

Three elements to implement the producer consumer model

1. Producers

2. Consumers

3. Queue

When to use the model:

There are obvious two categories of any, one task is responsible for production, the other is responsible for processing the production of data

The benefits of the model:

1, the realization of the producers and consumers decoupling and

2. Balance the productivity of producers with the ability of consumers to process data

Note: The producer consumer model is the solution to the problem of thinking is not technology. Can be implemented with processes and queues, or with other implementations.

From multiprocessing import joinablequeue,process
Import time
Import OS
Import Random

def producer (NAME,FOOD,Q):
For I in range (3):
res= '%s%s '% (food,i)
Time.sleep (Random.randint (1,3))
# Lost in the queue #
Q.put (RES)
Print (' \033[45m%s produced%s\033[0m '% (name,res))
# q.put (None)

DEF consumer (NAME,Q):
While True:
#从队列里取走
Res=q.get ()
If Res is none:break
Time.sleep (Random.randint (1,3))
Print (' \033[46m%s ate%s\033[0m '% (name,res))
Q.task_done ()

if __name__ = = ' __main__ ':
Q=joinablequeue ()
# producers '
P1=process (target=producer,args= (' Egon ', ' Bun ', Q,))
P2=process (target=producer,args= (' Yang June ', ' swill ', Q,))
P3=process (target=producer,args= (' Monkey teacher ', ' Xiang ', Q,))
# Consumers
C1=process (target=consumer,args= (' Alex ', Q,))
C2=process (target=consumer,args= (' WUPEIQIDSB ', Q,))
C1.daemon=true
C2.daemon=true

P1.start ()
P2.start ()
P3.start ()
C1.start ()
C2.start ()

P1.join ()
P2.join ()
P3.join ()
Q.join () #等待队列被取干净
# q.join () end means
# The code of the main process is finished---> (the producer is finished) + the data in the queue has been cleared. The meaning of the consumer does not exist

# print (' master ')

Producer-consumer models in Python