Python processes, threads, and co-routines

History of processes and threads

We all know that computers are made up of hardware and software. The CPU in the hardware is the core of the computer and it assumes all the tasks of the computer. Operating system is the software running on the hardware, is the manager of the computer, it is responsible for the management and allocation of resources, task scheduling. A program is a software that runs on a system that has some kind of functionality, such as a browser, a music player, and so on. Every time the execution of the program, will complete a certain function, such as browser to help us Open the Web page, in order to ensure its independence, we need a dedicated management and control of the execution of the program data structure-process control block. A process is a dynamic execution of a program on a data set. The process is generally composed of three parts: program, data set and process control block. The program we write is used to describe what the process is going to accomplish and how it is done, and the data set is the resource that the program needs to use in its execution, and the process control block is used to document the external characteristics of the process, to describe the process of its execution, and it can be used to control and manage the process, which is the only sign of

In the early operating system, the computer has only one core, the smallest unit of Process Execution program, and the task scheduling takes the time slice rotation preemptive method to carry on the process dispatch. Each process has its own piece of separate memory, which guarantees the isolation of the memory address space between the processes. With the development of computer technology, the process has a lot of drawbacks, one is the process of the creation, revocation and switching overhead is relatively large, the second is due to symmetric multiprocessor (symmetric multiprocessor (symmetricalmulti-processing) is called SMP, refers to a computer on the collection of a group of processors (Multi-CPU), the sharing of memory subsystem between each CPU and the bus structure, can meet multiple operating units, and multi-process parallel overhead is too large. The concept of threading is introduced at this time. A thread is also called a lightweight process, which is a basic CPU execution unit and the smallest unit in a program's execution, consisting of a thread ID, a program counter, a register collection, and a stack. The introduction of threads reduces the overhead of program concurrency and improves the concurrency performance of the operating system. Threads do not have their own system resources, they only have resources that are essential at run time. However, threads can share other resources owned by the process with other threads of the same process.

The relationship between a process and a thread

Threads are part of a process, threads run in process space, threads generated by the same process share the same memory space, and threads generated by the process are forced to exit and clear when the process exits. Threads can share all the resources owned by a process with other threads that belong to the same process, but they do not inherently have system resources and have only a bit of information that is essential in the run (such as program counters, a set of registers, and stacks).

Python threads

Threading is used to provide thread-related operations, which are the smallest unit of work in an application.

1. Threading Module

The threading module is built on the _thread module. The thread module processes and controls threads in a low-level, primitive way, while the threading module provides a more convenient API to process threads by encapsulating the thread two times.

Import threadingimport time   def worker (num): "" "    thread worker function    : return:    " " Time.sleep (1)    print ("The NUM is  %d", num)    return for   i in range:    t = Threading. Thread (target=worker,args= (i,), name= "t.%d"% i)    T.start ()

The code above creates 20 "foreground" threads, then the controller is handed over to the CPU,CPU according to the specified algorithm for scheduling, shard execution instructions.

Thread Method Description

T.start (): Activates thread,

T.getname (): Gets the name of the thread

T.setname (): Sets the name of the thread

T.name: Gets or sets the name of the thread

T.is_alive (): Determines whether a thread is active

T.isalive (): Determines whether a thread is active

T.setdaemon () is set to either a background thread or a foreground thread (default: False), and a Boolean value that sets whether the thread is a daemon thread and must be used after the start () method is executed. If it is a background thread, during the main thread execution, the background thread is also in progress, after the main thread executes, the background thread will stop whether or not it succeeds or not; If the foreground thread is executing, the foreground thread is also in progress, and after the main thread executes, waiting for the foreground thread to finish executing, the program stops

T.isdaemon (): Determines whether it is a daemon thread

T.ident: Gets the identifier of the thread. The thread identifier is a non-0 integer that is valid only after the start () method is called, otherwise it only returns none.

T.join (): Executes each thread one by one, execution continues, and the method makes multithreading meaningless

T.run (): The Run method that automatically executes thread objects after a thread is dispatched by the CPU

2, Line lock threading. Rlock and Threading.lock

The CPU then executes other threads because the threads are randomly dispatched, and each thread may execute only n execution. In order to ensure the accuracy of the data, the concept of lock was introduced. Therefore, the following problems may occur:

Example: Assuming that all elements of List A is 0, when one thread prints all the elements of the list backwards from the previous one, and the other thread modifies the list's elements from the back to 1, then the elements of the list will be partially 0 and some 1 in the output, resulting in inconsistent data. The presence of the lock solves the problem.

Import threadingimport time   globals_num = 0   lock = Threading. Rlock ()   def Func ():    lock.acquire ()  # get lock    global globals_num    globals_num + 1    time.sleep (1)    print (globals_num)    lock.release ()  # release lock for   I in Range (Ten):    t = Threading. Thread (Target=func)    T.start ()

3, Threading. The difference between Rlock and Threading.lock

Rlock is allowed to be acquire multiple times in the same thread. But lock does not allow this situation. If you use Rlock, then acquire and release must appear in pairs, that is, call n times acquire, must call the N-time release to really release the occupied locks.

Import Threadinglock = Threading. Lock ()    #Lock对象lock. Acquire () Lock.acquire ()  #产生了死琐. Lock.release () lock.release () Import Threadingrlock = Threading. Rlock ()  #RLock对象rLock. Acquire () Rlock.acquire ()    #在同一线程内, the program does not clog. Rlock.release () rlock.release ()

4, Threading. Event

The events of the Python thread are used by the main thread to control the execution of other threads, and the event provides three methods set, wait, clear.

Event handling mechanism: A global definition of a "flag", if the "flag" value is False, then when the program executes the Event.wait method is blocked, if the "flag" value is true, then the Event.wait method will no longer block.

Clear: Set "Flag" to False

Set: Sets "Flag" to True

Event.isset (): Determines whether the identity bit is ture.

Import Threading   def do:    print (' start ')    event.wait ()    print (' Execute ')   event_obj = Threading. Event () for I in range:    t = Threading. Thread (Target=do, args= (Event_obj,))    T.start ()   event_obj.clear () INP = input (' input: ') if InP = = ' true ':    Event_obj.set ()

When the thread executes, if flag is false, the thread blocks and the thread does not block when flag is true. It provides both local and remote concurrency.

5, Threading. Condition

A condition variable is always associated with some type of lock, which can be useful by using the default case or creating one, when several condition variables must be shared and the same lock. Locks are part of the Conditon object: There is no need to track them separately.

The condition variable is subject to the context Management protocol: The WITH statement block can obtain a connection to the lock before it is closed. Acquire () and release () call the appropriate method associated with the lock.

Other methods that are associated with the lock must be called, and the Wait () method releases the lock, which is blocked until another thread wakes it up with notify () or Notify_all (). Once awakened, wait () will regain the lock and return,

The condition class implements a Conditon variable. This Conditiaon variable allows one or more threads to wait until they are notified by another thread. If the lock parameter is given a non-null value, then he must be a lock or Rlock object, which is used to make the underlying lock. Otherwise, a new Rlock object is created to make the underlying lock.

Wait (Timeout=none): Waits for notification, or waits until the set time-out expires. When this wait () method is called, a RuntimeError exception is thrown if the thread calling it does not get a lock. Wati () After the lock is released, the other process that is called with the same condition is blocked until it wakes up with notify () or Notify_all (). Wait () can also specify a time-out.

If there is a waiting thread, the Notify () method wakes up a thread waiting for the Conditon variable. Notify_all () wakes all the threads waiting for the conditon variable.

Note: Notify () and Notify_all () do not release locks, that is, the threads are not immediately returned to their wait () calls after they are awakened. The lock ownership is discarded unless the thread calls notify () and Notify_all ().

In a typical design style, the condition variable is used to access some shared states with the lock Tongxu, and the thread repeatedly calls wait () before it gets to the state it wants. The thread that modifies the state calls notify () or Notify_all () when their state changes, and in this way the thread obtains as much as possible the desired state of the waiting person. Example: Producer-Consumer model,

Import Threadingimport timedef Consumer (cond): With    cond:        print ("Consumer before Wait")        cond.wait ()        Print ("Consumer after Wait")   def producer (cond): With    cond:        print ("producer before Notifyall")        Cond.notifyall ()        print ("producer after Notifyall")   condition = Threading. Condition () C1 = Threading. Thread (name= "C1", Target=consumer, args= (condition,)) C2 = Threading. Thread (name= "C2", Target=consumer, args= (condition,))   P = Threading. Thread (name= "P", Target=producer, args= (condition,)) C1.start   () time.sleep (2) C2.start () Time.sleep (2) P.start ( )

6. Queue Module

The queue is the pair of queues, it is thread-safe

For example, we go to McDonald's for dinner. There is a chef in the restaurant, the front desk is responsible for the kitchen to sell the food to customers, customers to the front desk to collect good food. The front desk here is the equivalent of our queue. To form a piping sample, the cook cooks the food through the front desk to the customer, so-called one-way queue

This model is also called producer-consumer models.

Import Queueq = queue. Queue (maxsize=0)  # Constructs an advanced presentation queue, maxsize specifies the queue length, which is 0 o'clock, which indicates that the queue length is unrestricted. Q.join ()    # Wait until the queue is Kong, perform other operations Q.qsize ()   # Returns the size of the queue (unreliable) q.empty ()   # returns True if the queue is empty, otherwise false (unreliable) q.full ()    # returns True when the queue is full, otherwise false (unreliable) q.put (item, Block=true, Timeout=none) # puts item in the  queue tail, The item must exist, and the parameter block defaults to True, indicating that when the queue is full, it waits for the queue to give the available location, false when it is non-blocking, and if the queue is full, it will raise queue. Full exception. Optional parameter timeout, which indicates the time at which the setting is blocked, and, after that,  if the queue cannot give the location to which it was placed, it raises the queue.　　　　　　　　　　　　　　　　　　　　　　Full exception Q.get (Block=true, Timeout=none) #   removes and returns a value for the header of the queue, the optional parameter block defaults to True, indicating that when the value is fetched, if the queue is empty, it is blocked, false, not blocked, If the queue is empty at this point, queue is raised. Empty exception. Optional parameter timeout, which indicates that the setting is blocked, and then, if the queue is empty, throws an empty exception. Q.put_nowait (item) # is   equivalent to put (Item,block=false) q.get_nowait () #    equivalent to get (Item,block=false)

The code is as follows:

#!/usr/bin/env pythonimport Queueimport threadingmessage = Queue.queue (Ten)    def producer (i): While    True:        Message.put (i)    def consumer (i): While    True:        msg = Message.get () for    I in range:    t = Threading. Thread (Target=producer, args= (i,))    T.start () for  I in range:    t = Threading. Thread (Target=consumer, args= (i,))    T.start ()

Then make yourself a thread pool:

# simple transfer threads to queue import Threadingimport timeimport queueclass threadingpool (): def __init__ (self,max_num = ten): Self.queue = queue. Queue (max_num) for I in Range (max_num): Self.queue.put (Threading. Thread) def getthreading (self): return Self.queue.get () def addthreading (self): Self.queue.put (Threadi Ng.    Thread) def func (p,i): Time.sleep (1) print (i) p.addthreading () if __name__ = = "__main__": P = Threadingpool () For I in range: thread = p.getthreading () t = thread (target = func, args = (p,i)) T.start () 

#往队列中无限添加任务import queueimport threadingimport Contextlibimport timestopevent = object () class ThreadPool (object): Def __ Init__ (self, max_num): self.q = queue. Queue () Self.max_num = max_num Self.terminal = False self.generate_list = [] Self.free_list = [] def run (self, func, args, Callback=none): "" "Thread pool performs a task:p Aram Func: Task function:p Aram args: any Functions Required Parameters:p Aram callback: A callback function executed after the failure or success of the task, the callback function has two parameters 1, the task function execution state, 2, the task function return value (default is None, that is: Do not execute the callback function): return: If the thread pool is already Terminates, returns true otherwise none "" "If Len (self.free_list) = = 0 and len (self.generate_list) < self.max_num:s        Elf.generate_thread () W = (func, args, callback,) Self.q.put (W) def Generate_thread (self): "" " Create a thread "" "T = Threading. Thread (Target=self.call) T.start () def call (self): "" Loop to get the task function and execute the task function "" "Current_ thread = Threading.currentthread self.generate_list. Append (current_thread) event = Self.q.get () # gets thread while event! = Stopevent: # Determines the number of threads fetched is not equal to the global variable Func, arguments, callback = event # split ancestor, get Execute function, parameter, callback function Try:result = func (*arguments) #                Row function status = True except Exception as e: # function execution Failed status = False                result = e If callback is not None:try:callback (status, result)  Except Exception as E:pass # self.free_list.append (current_thread) # Event  = Self.q.get () # Self.free_list.remove (Current_thread) with Self.work_state (): Event = Self.q.get () else:self.generate_list.remove (Current_thread) def close (self): "" "Close Thread to close the variable that transmits the global non-ancestor: return: "" "For I in Range (len (self.generate_list)): Self.q.put (stope VENT) def terminate (sELF): "" "abruptly closed Thread: return:" "" self.terminal = True while self.generate_list: Self.q.put (stopevent) self.q.empty () @contextlib. ContextManager def work_state (self): Self.fre E_list.append (Threading.currentthread) Try:yield Finally:self.free_list.remove (Threa Ding.currentthread) def work (i): print (i) return I +1 # return to callback function Def callback (ret): print (ret) pool = ThreadPool (Ten) F or item in range: Pool.run (func=work, args= (item,), Callback=callback) pool.terminate () # pool.close ()

Python process

Multiprocessing is a multi-process Management Pack for Python, and threading. Thread is similar.

1. Multiprocessing Module

By replacing threads with subprocesses directly from the side using the Gil, the multiprocessing module allows programmers to make full use of the CPU on a given machine. In multiprocessing, you create a process object by creating it, and then call its start () method,

From multiprocessing import Process  def func (name):    print (' Hello ', name)   if __name__ = = "__main__":    p = Process (target=func,args= (' Zhangyanlin ',))    P.start ()    p.join ()  # Wait for the process to finish executing

It is best to avoid sharing data as much as possible when using concurrent designs, especially when using multiple processes. If you really need to share data, multiprocessing provides two ways.

(1) Multiprocessing,array,value

The data can be stored in a shared memory map with value or array, as follows:

From multiprocessing import array,value,process  def func (A, b):    a.value = 3.333333333333333 for    i in range ( Len (b)):        b[i] =-b[i]    if __name__ = = "__main__":    num = Value (' d ', 0.0)    arr = Array (' I ', Range (one))        C = Process (target=func,args= (Num,arr))    d= process (target=func,args= (Num,arr))    C.start ()    D.start ()    c.join ()    d.join ()      print (num.value) for    i in arr:        print (i)

Output:

3.1415927

[0,-1,-2,-3,-4,-5,-6,-7,-8,-9]

When Num and arr are created, the "D" and "I" parameters are created by the typecodes used by the array module: "D" represents a double-precision floating-point number, and "I" represents a signed integer that will be thread-safe to handle.

The ' i ' parameter in Array (' I ', Range (10)):

' C ': Ctypes.c_char ' u ': Ctypes.c_wchar ' B ': Ctypes.c_byte ' B ': ctypes.c_ubyte

' H ': Ctypes.c_short ' h ': Ctypes.c_ushort ' i ': Ctypes.c_int ' i ': ctypes.c_uint

' L ': ctypes.c_long, ' l ': Ctypes.c_ulong ' F ': Ctypes.c_float ' d ': ctypes.c_double

(2) Multiprocessing,manager

Manager () returned by Manager provides list, dict, Namespace, Lock, Rlock, Semaphore, Boundedsemaphore, Condition, Event, Barrier, Queue, The Value and array type support.

From multiprocessing import process,managerdef F (d,l):    d["name"] = "Zhangyanlin"    d["age"] =    d["Job"] = " Pythoner "    l.reverse ()  if __name__ = =" __main__ ": With    Manager () as man:        d = man.dict ()        L = man.list ( Range (Ten)          p = Process (target=f,args= (d,l))        P.start ()        p.join ()          print (d)        print (L)

Output:

{0.25:none, 1: ' 1 ', ' 2 ': 2}

[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

The Server process Manager is more flexible than the shared memory because it can support any object type. In addition, a separate manager can be shared between different computers on the network through a process, but he is slower than shared memory.

2, Process pool (Using a pool of workers)

The pool class describes a worker process pools, and he has several different ways to get tasks to unload worker processes.

A process sequence is maintained internally by the process pool, and when used, a process is fetched in the process pool, and the program waits until a process is available in the process pool sequence if there are no incoming processes available for use.

We can create a process pool with the pool class and expand the submitted task to the process pool. Cases:

#applyfrom  multiprocessing Import poolimport time  def F1 (i):    time.sleep (0.5)    print (i)    return i +  if __name__ = = "__main__":    pool = Pool (5) for    I in range (1,31):        pool.apply (func=f1,args= (i,))  #apply_asyncdef F1 (i):    time.sleep (0.5)    print (i)    return i + 100def F2 (ARG):    print (ARG)  if __ name__ = = "__main__":    pool = Pool (5)    for I in Range (1,31):        pool.apply_async (func=f1,args= (i,), callback =F2)    pool.close ()    pool.join ()

A process Pool object can control what work in the worker process pool can be committed, it supports asynchronous results for timeouts and callbacks, and has a map-like implementation.

Processes: The number of worker processes used, if processes is none then use Os.cpu_count () to return the quantity.

Initializer: If initializer is none, then each worker process will call initializer (*initargs) at the beginning.

Maxtasksperchild: The number of tasks that can be completed before the worker process exits, and after completion, replace the original process with a heart-working process to allow idle resources to be freed. Maxtasksperchild default is None, which means that the pool exists as long as the worker process is alive.

Context: Use multiprocessing in general when developing a work process. Pool () or a context object's pool () method to create one, both methods are appropriately set up for the context

Note: The method of the pool object can only be called by the process that created the pool.

New in version 3.2:maxtasksperchildnew in version 3.4:context

Method of the process pool

Apply (func[, args[, Kwds])

: The Func function is called with the ARG and Kwds parameters, and the result is blocked until it returns, for this reason, Apply_async () is more suitable for concurrent execution, and the Func function is only run by one process in the pool.

Apply_async (func[, args[, kwds[, callback[, Error_callback]]]): A variant of the Apply () method, which returns a result object. If callback is specified, then callback can receive a parameter and be called, and when the result is ready for the callback, the callback is called, and when the call fails, the callback is replaced with Error_callback. Callbacks should be completed immediately, otherwise the thread that processed the result will be blocked.

Close (): Prevents more tasks from being submitted to the pool, and the worker process exits when the task is completed.

Terminate (): Stops the worker process immediately, regardless of whether the task is completed or not. When the pool object process is garbage collected, terminate () is called immediately.

Join (): The wait worker thread exits before calling join () and must call Close () or terminate (). This is because the terminated process needs to be called by the parent process wait (join equals wait), otherwise the process will become a zombie process.

Map (func, iterable[, chunksize]) ¶map_async (func, iterable[, chunksize[, callback[, Error_callback]]) ¶imap (func, iterable[, chunksize]) ¶imap_unordered (func, iterable[, chunksize]) Starmap (func, iterable[, chunksize]) ¶starmap_ Async (func, iterable[, chunksize[, callback[, Error_back]])

Python co-process

The operation of the thread and process is triggered by the program to trigger the system interface, the final performer is the system, and the operation of the coprocessor is the programmer.

The significance of the existence of the process: for multi-threaded applications, the CPU by slicing the way to switch between threads of execution, thread switching takes time (save state, next continue). , only one thread is used, and a code block execution order is specified in one thread.

Application scenario: When there are a large number of operations in the program that do not require the CPU (IO), it is suitable for the association process;

The event loop is the control point for the execution of the process, and you need to use it if you want to perform the co-process.

The event loop provides the following features:

Register, execute, Cancel delay call (Async function)

Create client and server protocols (tools) for communication

Create child processes and protocols that communicate with other programs (tools)

To feed a function call into the thread pool

Example of a process:

Import Asyncio   Async def cor1 ():    print ("COR1 start")    await Cor2 ()    print ("COR1 end")   async def Cor2 ():    print ("COR2")   loop = Asyncio.get_event_loop () loop.run_until_complete (Cor1 ()) Loop.close ()

The last three lines are the focus.

Asyncio.get_event_loop (): Asyncio start the default event loop

Run_until_complete (): This function is blocking execution, knowing that all asynchronous functions are executed,

Close (): Closes the event loop.

1, Greenlet

Import Greenletdef fun1 ():    Print ("page")    gr2.switch ()    print ("")    Gr2.switch ()  def fun2 ():    Print ("Gr1.switch")    print ("Gr1")    = Greenlet.greenlet (fun1) GR2 = Greenlet.greenlet (fun2) Gr1.switch ()

2, Gevent

Gevent belongs to the third party module need to download the installation package

PIP3 Install--upgrade pip3pip3 Install Gevent

Import Geventdef fun1 ():    print ("www.baidu.com")   # First step    gevent.sleep (0)    print ("End the Baidu.com")  # Step three  def fun2 ():    print ("www.zhihu.com")   # Step two    gevent.sleep (0)    print ("End th Zhihu.com ")  # Fourth Step  gevent.joinall ([    gevent.spawn (FUN1),    gevent.spawn (fun2),])

Automatic switching of IO operation encountered:

Import geventimport requestsdef func (URL):    print ("Get:%s"%url)    gevent.sleep (0)    date =requests.get (URL)    ret = date.text    print (Url,len (ret)) Gevent.joinall ([    gevent.spawn (func, ' https://www.pythontab.com/') ,    Gevent.spawn (func, ' https://www.yahoo.com/'),    gevent.spawn (func, ' https://github.com/'),])

Article turned from: http://www.cnblogs.com/aylin/p/5601969.html

