Python's 7 process thread and a detailed description of the association

Preface: Diagram of Threads and processes

From the point of view, in each application execution process, will produce a master process and the main thread to complete the work, when we need concurrent execution, it will be through the main process to generate a series of sub-processes (and then through the child process to produce a series of child threads) to make different CPU calls, so as to achieve the concurrency effect. However, it is important to note that in general, each process is independent of each other.

The Gil Global interpreter lock is unique in Python, not in Java or C #, what is his main role? We all know that the execution of the program is the smallest unit is a thread, when the CPU1 through the process to call the thread (just when the CPU calls), can only be polled to invoke a thread in a process, the thread does not carry out concurrent execution, That means that at one time each CPU can only do a single thread in a process to do a job.

　In our general program, if we do not deliberately create processes and threads, then our program is executed in sequence, and when we create processes and threads, we have the effect of concurrent execution.

Process

Advantage: Multiple CPUs can be used at the same time for multiple operations

Cons: Re-opening up memory space, very resource-intensive

Number: General and CPU number of the same

Place of use: generally computationally intensive

Thread

Pros: Shared memory (in-process), I/O operations enable concurrent execution

Cons: Seizing resources, switching contexts is time consuming

Number: Generally depending on the situation

Places of Use: I/O intensive

I. Process

Definition of the process

Process is a computer program on a data set on a running activity, the system is the basic unit of resource allocation and scheduling. In fact, the process is an instance of program execution, where the program is not executed, only by creating a process to complete the operation of the program. For example: I want to cook now, first I pick up the chopper, then I go to cut vegetables, then fire, stir fry. Cooking is actually a program, I take a knife, cut vegetables, fire, stir fry can be seen as a process, they in the execution flow of the program executed in order to complete an operation.

1. Creating processes (Process Class)

For a program we write, the default will have a master process and the main thread from the top to the small to execute the code, if one encounters to create the process and thread, then the main process will create the process and the thread, (then the child process and the child threads created by themselves to execute the execution of his code), After creation, there are two actions, one is to wait for the child process or sub-thread to complete after the completion of the program, the other is that when my main process is complete, the program will be finished immediately, regardless of your child process or the sub-line threads not complete.

# To do experiments under Windows, the first sentence must be preceded by the IF __name__ = = "__main__": # Create a process, # The name of the function behind the target is the one that the process is going to execute. # args followed by a tuple, The parameter P = multiprocessing is required to represent the function behind the target. Process (Target=foo, args= (1,))        P.join (5)   # When you finish executing this subprocess and then go to another process, the parameter 5 represents the maximum time to wait for the child process to execute, and the default is s# Daemon is whether the master process waits for the child process to finish before it ends, and the default is to wait for # True to represent not waiting for # False to wait for P.daemon = True     P.start ()   # to start a child process

1 # The following code shows that the result is empty because the program ends after the main process ends 2 # does not go to execute Foo function 3 import multiprocessing 4 import time 5  6 def foo (args): 7     # this is A function to be executed through a child process 8     time.sleep (3)   # delay three seconds 9     print (args) x     if __name__ = = "__main__":     p = Multiprocessing. Process (Target=foo, args= (1,))     P.daemon = True # does not wait for the end     of the child process P.start ()   15 16 17 # The following code executes the result of 1 Because the value of daemon is false, the main process waits for the child process to finish executing foo before it ends the program. Import multiprocessing19 import time20 def foo (args):     This is a function to be executed through a child process.     Time.sleep (3)   # Delay three sec (     args) +     if __name__ = = "__main__":     p = Multiprocessing. Process (Target=foo, args= (1,))     P.daemon = False # does not wait for the end of the child process     P.start ()

Case One daemon

# when there is no join, the input result is basically the simultaneous output of the 123456789import multiprocessingimport timedef foo (args): # This is a function to be executed through a child process time.sleep (1) Print (args) if __name__ = = "__main__": For I in range:        p = multiprocessing. Process (Target=foo, args= (i,))        P.start () #有join的时候, he is an output, because the join represents the execution of the other process import after the child process has finished executing Multiprocessingimport timedef foo (args): # This is a function to be executed through a child process Time.sleep (1) print (args) if __name__ = = "__main__": For I In Range:        p = multiprocessing. Process (Target=foo, args= (i,))        P.start ()        P.join (2)

Case Two Join

# This code will not output any value, and when the program executes 1s it will end because the join default waits for 1s, but your child process will need 10s, so the child process has not finished executing the main process to end the import multiprocessingimport Timedef foo (args): # This is a function to be executed through a subprocess Time.sleep () print (args) if __name__ = = "__main__":    p = multiprocessing. Process (Target=foo, args= (1,))    P.daemon = True    p.start ()    p.join (1)

Case Three Join 　

2. Process pools (pool module)

What is a process pool? The popular point is the container of the process, when we write the program, we can not come to a program, we go to create a process, process is very resource-intensive, so we define a process in advance of the container (the number of processes is fixed), when our program needs to automatically go to the process pool central to take, If the number of child processes in the process pool has been exhausted, we will only be able to continue to use it until other programs have been released.

if __name__ = = "__main__": # Create Process Pool Proc_pool = multiprocessing. Pool (5) # The following two are all ways to use the process pool # apply: He uses the Join method internally, and after each child process is done, the next subprocess is used # Apply_async: He doesn't use the Join method internally, so it's the execution of all the child processes concurrently    proc_pool.apply ()    Proc_pool.apply_async ()

1 # From the result it can be seen that each child process is finished before it prints out the last child process creation completed 2 import multiprocessing 3 import time 4  5 def foo (S1): 6     time.sleep (1) 7< C2/>print (S1) 8 if __name__ = = "__main__": 9     # Create a process pool with the number of processes 510 proc_pool     = multiprocessing. Pool (5) One     for I in range:         # 10 child processes are created, each subprocess executes the Foo function, and the passed arguments are i13         proc_pool.apply (foo, args= (i,)) 14     print ("Child process Creation complete") 15 16 output: 17 018 119 220 321 422 523 624 725 826 927 child process creation complete

Case One apply

# The result is that the process is created first, and as you can see from the execution results, the Apply_async function causes all child processes to execute concurrently, after which the join function will cause the main process to wait for the child process to complete before closing the program import Multiprocessingimport time# executed function def foo (S1):    time.sleep (1) Return s1# callback function def foo2 (S1):p rint (S1) If __name__ = "__main__": # Create Process Pool, The number of processes is 5proc_pool = multiprocessing. Pool (5) for I in Range (10): # 10 sub-processes are created, each subprocess executes the Foo function, the passed parameter is I, the return value of the Foo function as a parameter to Foo2, and then the Foo2 function Proc_pool.apply_async (foo , args= (i,), Callback=foo2) print ("Child process Creation complete") # Close Process Pool    proc_pool.close () # Wait for the child process to finish executing and return to    Proc_pool.join () Output: Child process creation completed 012345678

Case 2 APPLY_AYSNC

3. Sharing between processes

Processes are inherently independent and mutually exclusive, and if you really want to communicate between processes, there are two ways.

<1>. Array

<2>. Manage module to create special data types

Import Multiprocessingimport Multiprocessingdef F1 (S1, dic):    dic[s1] = s1if __name__ = = "__main__": # Create a Manage object manage = multiprocessing. Manager () # Creates a special type of dict from manage for use between processes using DIC = Manage.dict () print ("DiC before Modification:", dic) for I in range:        p = Multiprocessing. Process (TARGET=F1, args= (i, DIC))        P.start ()        p.join () print ("Modified dic:", DIC) results: No prior modification of DIC: {} modified dic: {0:0, 1:1, 2:2, 3:3, 4:4, 5:5, 6:6, 7:7, 8:8, 9:9}

Two. Threads

A thread is the execution unit at the bottom of a program, and he is essentially a process, but a process that is more nuanced and is used to execute the program.

1. Create a thread

# thread creation and process creation are similar, because the thread is the form of the process # The following methods and processes are the same way, is to turn daemon into setdaemon just if __name__ = = "__main__": # The Create thread Foo function is the function to be executed with the child thread, and args is the passed parameter, thread = threading. Thread (Target=foo, args= (1,)) # The    maximum time to start a thread Thread.Start () # child thread, etc. thread.join (5) # Sets whether to wait for a child thread to finish after the main process completes Default is no wait for Thread.setdaemon (True)

1 # The following code is not the result, because the thread and process are not the same, the thread default is not waiting for the child thread of 2 import threading 3 Import time 4 # function executed 5 def foo (S1): 6     time.sleep (1) 7< C1/>print (S1) 8  9 If __name__ = = "__main__": ten     thread = threading. Thread (Target=foo, args= (1,)) one     Thread.Start () 12 13 # Modify the following code to show the result, import threading15 import Time16 # functions executed def foo (S1):     time.sleep (1)     print (s1) if __name__ = = "__main__":     thread = Threading. Thread (Target=foo, args= (1,))     Thread.setdaemon (False)     Thread.Start ()

Case One Setdaemon

1 Import Threading 2 Import time 3  4 # functions performed 5 def foo (S1): 6     time.sleep (1) 7     print (S1) 8  9 If __name_ _ = = "__main__": Ten for     I in range (5):         thread = Threading. Thread (Target=foo, args= (i,))         Thread.Start ()         Thread.Join (2)

Case Two Join

2. Rlock Module

Rlock module from the name can be seen as a lock module, we all know that threads are shared memory, so when two threads modify a value at the same time, there will be dirty values (that is, we do not expect the value), because we do not know which thread modification is valid, so this module was born, When we want to modify a value, we can use the lock module to lock the value.

1 # In fact, this example does not show the confusion of data .... 2 # It's just a simple word. How to use the Rlock Module 3 import Threading 4 Import time 5  6 # Create a global variable to be modified with a thread 7 num = [] 8 # Create a Lock object 9 lock = Threading. Rlock () 10 # functions performed by Def foo (S1):     # plus lock     # lock.acquire ()     global num15     num.append (S1)     print ( NUM)     # release Lock     # lock.release () + if __name__ = = "__main__":     -I in range (+):         thread = Threading. Thread (Target=foo, args= (i,))         Thread.Start ()     print (num)

Rlock

3. Event Module

The event module is actually the meaning of pause, and when we use this module, the thread will stop here, and when we set the corresponding value, we will continue to execute.

1 Import Threading 2 Import time 3  4 # Create a global variable to use the thread to modify it 5 num = [] 6 # Create a lock object 7 lock = Threading. Rlock () 8 event = Threading. Event () 9 # function performed by Def foo (S1):     # plus lock     lock.acquire ()     # thread on this pause (red light)     event.wait ()     NUM16     num.append (S1)     print (num)     # release lock     lock.release () if __name__ = = "__main__": For     I In range (5):         thread = Threading. Thread (Target=foo, args= (i,))         Thread.Start ()     event.clear ()   # set to red light     INP = input ("Enter Q to continue:")     if InP = = ' Q ':         if the input is Q, the status of the event will change, continue to execute         event.set () 29 30 31 Result Output 32 input true continue: q33 [0]34 [0, 1]35 [0, 1, 2]36 [0, 1, 2, 3]37 [0, 1, 2, 3, 4]

the state after Event+lock

1 Import Threading 2 Import time 3  4 # Create a global variable to use the thread to modify it 5 num = [] 6 # Create a lock object 7 lock = Threading. Rlock () 8 event = Threading. Event () 9 # function performed by Def foo (S1):     # plus lock     # lock.acquire () #     thread on this pause (red light)     event.wait ()     NUM16     num.append (S1)     print (num)     # release Lock     # lock.release () if __name__ = = "__main__":     I in range (5):         thread = Threading. Thread (Target=foo, args= (i,))         Thread.Start ()     event.clear ()   # set to red light     INP = input ("Enter Q to continue:")     if InP = = ' Q ':         # # If the input is Q, change the waiting state of the event, and proceed with the execution of         Event.set () 29 30 output: 31 input Q continue: q32 [0]33 [0, 2]34 [0, 2 , 1]35 [0, 2, 1, 4]36 [0, 2, 1, 4, 3]

Event Model

4. Producer Consumer Model (Queue module)

The producer consumer model actually says is the queue, the queue we only need to remember the FIFO first.

# import the module of the queue imports queue# create a queue with a queue length of up to 5obj = queues. Queue (5) # Gets the value from the queue and waits for Obj.get () # to get the value from the queue if the queue is empty, or discard the value (do not wait) obj.get_nowait () # to upload a value to the queue Obj.put ("value")

5. Thread pool

There is no way to create a thread pool by default in Python, so here is a summary of the two methods of the Wupeiqi teacher, the address of the method is as follows

　　　　

Some parts of this code are difficult to understand, the main reason is that the previously written code is executed sequentially, and for threads and processes can be executed concurrently, so for the execution of the flow still need to be noted.

1 Import Queue 2 import threading 3 Import time 4  5 class Threadpool:6     def __init__ (self, max_num): 7 self         . Threadqueue = queue. Queue (Max_num) 8 for         I in range (Max_num): 9 self             . Threadqueue.put (Threading. Thread)     -def get_thread (self): return to self         . Threadqueue.get ()     def add_thread (self): + self         . Threadqueue.put (Threading. Thread)-def func (pool, args):     time.sleep (2)     print (args)     pool.add_thread ()

thread pool implementation--simple method

  1 #-*-Coding:utf-8-*-2 # Zhou 3 # 2017/7/5 4 5 Import threading 6 Import queue 7 Import time 8 9 # list exit flag bit         Stopevent = Object () One class threadpool:13 def __init__ (self, max_num): 14 # Create an empty queue to hold tasks instead of threads 15 SELF.Q = queue.          Queue () 16 # Set the number of idle threads to 0 self.free_list = [] 18 # Number of threads already created self.generate_list = [] 20 # Maximum number of threads created self.max_num = max_num 22 # Create Task List empty self.task = [] Self.ter  Minal_flag = False + def apply (self, target, args, Callback=none): 27 # Gets the task list.          Args, callback,) # print (' * * * ', args) 30 # Add Task List to queue Self.q.put (Task) 32 # to execute 33 If Len (self.free_list) = = 0 and len (self.generate_list) < self.max_num:34 # If there are no idle threads and the number of threads created is less than the maximum Number of threads, create a thread of Self.generate_thread () Generate_thread (self): $ t = Threading. Thread (TarGet=self.run) T.start () (self): Current_thread = Threading.currentthread 43             Self.generate_list.append (current_thread) The event = Self.q.get () while event! = stopevent:46             # is the task, untie the task Pack, perform the task func1, argument, Func2 = event # print ("+ +", argument) 49                 TRY:50 ret = func1 (*argument), state = True except Exception as e:53                     State = False FUNC2 ret = e If the is not none:56 try:57  Func2 (State, ret) except Exception as e:59 Pass if Not self.terminal_flag:61 self.free_list.append (current_thread) event = Self.q.get ()         Self.free_list.remove (current_thread) else:65 event = Stopevent 66            else:67 # is not a task, remove the Self.generate_list.remove (current_thread) (+) def close (self): # Stopevent as a follow-up Ring end of the flag, how many threads will give him the number of flags to be created, num = Len (self.generate_list), while num:74 Self.q.put (stopeve NT) num-= 1 terminal def (self): Self.terminal_flag = True + while Self.gene rate_list:80 self.q.put (stopevent) Bayi # self.close () Self.q.empty () 83 # Execute function def foo (s 1): Time.sleep (0.5) s1 print () 87 # callback function F2 (state, S2): print (S2), if __name__ = "__ma In__ ": 92 # Create a thread pool ThreadPool (5) 94 for I in Range (40): 95 # application thread pool # print (' ___ ') , i) pool.apply (Target=foo, args= (i,)) 98 Time.sleep (4) pool.terminal () 100 101

thread pooling implementation--a complex approach

Three. Co-process

What is the co-process? The process is actually micro-threading, for example, the association is generally used in web page requests above, using the Gevent to import module, the following a simple use example

1 #-*-Coding:utf-8-*-2 # Zhou 3 # 2017/7/5 4 import gevent 5 Import requests 6  7 def F1 (URL): 8     requests.get (U RL) 9 Gevent.joinall ([     gevent.spawn (F1, ""),     gevent.spawn (F1, ""), 13]14)

How to use the co-process

Four. Context switch (CONTEXTLIB)

In fact, this context switch and adorner is a bit similar, but also before and after an operation to add a bit of operation.

The following code executes the process

　　

Import Contextlib@contextlib.contextmanagerdef File_open (file_name, mode):    f = open (file_name, mode) Try:yield Ffinally:        f.close () with File_open (' Te ', ' R ') as Obj_f:print (Obj_f.read ())