Python task execution thread, process, and coroutine, python

Python task execution thread, process, and coroutine, python

I. threads

The thread is the minimum unit for executing tasks in the program. The Threading module provides related operations. The thread is suitable for intensive IO operations.

1 #! /Usr/bin/env python 2 #-*-coding: UTF-8-*-3 import threading 4 import time 5 6 def show (arg): 7 time. sleep (1) 8 print 'thread' + str (arg) 9 10 for I in range (10): 11 t = threading. thread (target = show, args = (I,) 12 t. start () 13 14 print 'main thread stop'Basic thread usage

The above Code creates 10 "foreground" threads, and then the controller is handed over to the CPU, the CPU Schedules according to the specified algorithm, and executes commands in parts.

More operations are as follows:

1. The start thread is ready. Wait for CPU scheduling 2. Set setName to thread name 3. getName to get thread name 4. Set setDaemon to background thread or foreground thread (default) 5. If it is background thread, during the execution of the main thread, the background thread is also in progress. After the main thread completes execution, the background thread stops 6 if it is a foreground thread and the main thread is in the process of execution, the front-end thread is also running. After the main thread is executed, wait until the front-end thread is also executed, the program stops 7 join and executes each thread one by one. After the execution is completed, the program continues to run, this method makes multithreading meaningless. 8. After the run thread is scheduled by the cpu, the run method of the thread object is automatically executed.View Codeimport threadingimport time class MyThread (threading. thread): def _ init _ (self, num): threading. thread. _ init _ (self) self. num = num def run (self): # define the print ("running on number: % s" % self. num) time. sleep (3) if _ name _ = '_ main _': t1 = MyThread (1) t2 = MyThread (2) t1.start () t2.start ()Custom Thread class

Since random scheduling is performed between threads, and each thread may only execute n executions, confusion may occur when multiple threads modify the same data at the same time, A thread lock occurs-a thread is allowed to perform operations at the same time. The Code is as follows:

1 #! /Usr/bin/env python 2 # coding: UTF-8 3 4 import threading 5 import time 6 7 gl_num = 0 8 9 lock = threading. RLock () # define lock 10 11 def Func (): 12 lock. acquire () # Use the lock 13 global gl_num14 gl_num ++ = 115 time. sleep (1) 16 print gl_num17 lock. release () # unlock 18 19 for I in range (10): 20 t = threading. thread (target = Func) 21 t. start ()View Code

 

The mutex lock allows only one thread to change data at the same time, while Semaphore allows a certain number of threads to change data at the same time. For example, if there are three pits in the toilet, only three persons can go to the toilet at most, the people later can only wait for someone to come in.

1 import threading, time 2 3 def run (n): 4 semaphore. acquire () 5 time. sleep (1) 6 print ("run the thread: % s" % n) 7 semaphore. release () 8 9 if _ name _ = '_ main _': 10 11 num = 012 semaphore = threading. boundedSemaphore (5) # Up to five threads can run at the same time 13 for I in range (20): 14 t = threading. thread (target = run, args = (I,) 15 t. start ()Mutual lock Removal

Event)

The events of the python thread are used by the main thread to control the execution of other threads. The events mainly provide three methods: set, wait, and clear.

Event processing mechanism: a global "Flag" is defined. If the "Flag" value is False, when the program executes the event. the wait method is blocked. If the "Flag" value is True, the event. the wait method is no longer blocked.

• Clear: Set "Flag" to False
• Set: set "Flag" to True

#! /Usr/bin/env python #-*-coding: UTF-8-*-import threading def do (event): print 'start' event. wait () print 'execute 'event_obj = threading. event () for I in range (10): t = threading. thread (target = do, args = (event_obj,) t. start () event_obj.clear () indium = raw_input ('input: ') if indium = 'true': event_obj.set ()Event instance

Condition to make the thread wait. Only when the condition is met will n threads be released

Import threading def run (n): con. acquire () con. wait () print ("run the thread: % s" % n) con. release () if _ name _ = '_ main _': con = threading. condition () for I in range (10): t = threading. thread (target = run, args = (I,) t. start () while True: indium = input ('>>>') if indium = 'q': break con. acquire () con. using Y (int (indium) con. release ()View Code

 

Timer. The operation is performed after n seconds.

From threading import Timer def hello (): print ("hello, world") t = Timer (1, hello) t. start () # after 1 seconds, "hello, world" will be printedView Code

Ii. Process

Process Creation consumes memory and requires careful creation. Multiple processes are suitable for computing-intensive scenarios.

From threading import Timer def hello (): print ("hello, world") t = Timer (1, hello) t. start () # after 1 seconds, "hello, world" will be printedBasic use of processes

By default, the data of a process is not shared (the thread shares the memory data), so the overhead is large and data cannot be shared by default.

#! /Usr/bin/env python # coding: UTF-8 from multiprocessing import Processfrom multiprocessing import Manager import time li = [] def foo (I): li. append (I) print 'say Hi', li for I in range (10): p = Process (target = foo, args = (I,) p. start () print 'ending', liData cannot be shared by default # method 1, Arrayfrom multiprocessing import Process, Arraytemp = Array ('I', [,]) def Foo (I ): temp [I] = 100 + I for item in temp: print I, '----->', item for I in range (2): p = Process (target = Foo, args = (I,) p. start () # Method 2: manage. dict () shared data from multiprocessing import Process, Manager manage = Manager () dic = manage. dict () def Foo (I): dic [I] = 100 + I print dic. values () for I in range (2): p = Process (target = Foo, args = (I,) p. start () p. join ()Data sharing method 'C': ctypes. c_char, 'U': ctypes. c_wchar, 'B': ctypes. c_byte, 'B': ctypes. c_ubyte, 'H': ctypes. c_short, 'H': ctypes. c_ushort, 'I': ctypes. c_int, 'I': ctypes. c_uint, 'L': ctypes. c_long, 'L': ctypes. c_ulong, 'F': ctypes. c_float, 'D': ctypes. c_doubleType table from multiprocessing import Process, queuesimport multiprocessingdef f (I, q): print (I, q. get () if _ name _ = '_ main _': q = queues. queue (ctx = multiprocessing) q. put ('h1 ') q. put ('h2 ') q. put ('h3 ') for I in range (3): p = Process (target = f, args = (I, q,) p. start ()Data sharing through queues

 

When a process is created (not used), the shared data is obtained to the sub-process. After the sub-process is executed, the original value is assigned. Therefore, the process is also locked.

#! /Usr/bin/env python #-*-coding: UTF-8-*-from multiprocessing import Process, Array, RLockdef Foo (lock, temp, I ): "add 0th to 100" "lock. acquire () temp [0] = 100 + I for item in temp: print I, '----->', item lock. release () lock = RLock () temp = Array ('I', [11, 22, 33, 44]) for I in range (20 ): p = Process (target = Foo, args = (lock, temp, I,) p. start ()Process lock

Process pool

A process sequence is maintained in the Process pool. When used, a process is obtained from the process pool. If there is no usable process in the process pool sequence, the program will wait, until there are available processes in the process pool.

There are two main methods in the process pool:

• Apply
• Apply_async

#! /Usr/bin/env python #-*-coding: UTF-8-*-from multiprocessing import Process, Poolimport time def Foo (I): time. sleep (2) return I + 100 def Bar (arg): print arg pool = Pool (5) # print pool. apply (Foo, (1,) # print pool. apply_async (func = Foo, args = (1 ,)). get () for I in range (10): pool. apply_async (func = Foo, args = (I,), callback = Bar) print 'end' pool. close () pool. join () # Shut down the processes in the process pool after they are executed. If comments are made, the program is closed directly.View Code

Apply_async is non-blocking, apply is blocking, and apply_async has one more parameter, which is the callback function.

More Methods

• Close () closes the pool so that it does not accept new tasks. • Terminate () ends the working process and does not process unfinished tasks. • The join () primary process is blocked and waits for the sub-process to exit. The join method should be used after close or terminate.View Code

Iii. coroutine

The operations of threads and processes are system interfaces triggered by programs, and the executors are systems. The operations of coroutines are programmers.

The meaning of coroutine: For multi-threaded applications, the CPU uses slices to switch the execution between threads. It takes time to switch the threads (Save the status and continue next time ). Coroutine, only one thread is used to specify the execution sequence of a code block in one thread.

Application scenarios of coroutine: when a program contains a large number of operations that do not require CPU (I/O), it is suitable for coroutine. Simply put, it is to optimize the tasks executed by the thread, and execute them in slices, for example, when a thread makes a network request, there will be a delay. During this time, the thread is waiting, and the wait time is used for other tasks.

Basic coroutine Module

#! /Usr/bin/env python #-*-coding: UTF-8-*-from greenlet import greenlet def test1 (): print 12 gr2.switch () print 34 gr2.switch () def test2 (): print 56 gr1.switch () print 78 gr1 = greenlet (test1) gr2 = greenlet (test2) gr1.switch ()Greenlet

Well-encapsulated collaborative modules

Import gevent def foo (): print ('running in foo') gevent. sleep (0) print ('explicit context switch to foo again ') def bar (): print ('explicit context to bar') gevent. sleep (0) print ('implicit context switch back to bar') gevent. joinall ([gevent. spawn (foo), gevent. spawn (bar),])Gevett

IO Automatic Switch

From gevent import monkey; monkey. patch_all () import geventimport urllib2def f (url): print ('get: % s' % url) resp = urllib2.urlopen (url) data = resp. read () print ('% d bytes encoded ed from % s. '% (len (data), url) gevent. joinall ([gevent. spawn (f, 'https: // www.python.org/'), gevent. spawn (f, 'https: // www.yahoo.com/'), gevent. spawn (f, 'https: // github.com/'),])View Code