Break through the python defects and implement several custom thread pools as well as introduction of processes, threads, and coroutines.

Break through the python defects and implement several custom thread pools as well as introduction of processes, threads, and coroutines.
Python thread

Threading is used to provide thread-related operations. A thread is the smallest unit of operation in an application.

#!/usr/bin/env python# -*- coding:utf-8 -*-import threadingimport time  def show(arg):    time.sleep(1)    print 'thread'+str(arg)  for i in range(10):    t = threading.Thread(target=show, args=(i,))    t.start()  print 'main thread stop'

　　

The above Code creates 10 "foreground" threads, and then the controller is handed over to the CPU, the CPU Schedules according to the specified algorithm, and executes commands in parts.

More methods:

• The start thread is ready for CPU scheduling.
• SetName indicates the thread name.
• GetName: Get the thread name
• SetDaemon is set to the background thread or foreground thread (default)
  If it is a background thread, the background thread is also running during the main thread execution process. After the main thread is executed, the background thread stops no matter whether it is successful or not.
  If it is a foreground thread, the foreground thread is also running during the main thread execution. After the main thread is executed, the program stops after the foreground thread is executed.
• Join executes each thread one by one, and continues to run after execution. This method makes multithreading meaningless.
• After the run thread is scheduled by the cpu, the run method of the thread object is automatically executed.

Thread lock

Since threads are randomly scheduled, and each thread may only execute n executions, the CPU then executes other threads. Therefore, the following problems may occur:

No lock used

#!/usr/bin/env python#coding:utf-8   import threadingimport time   gl_num = 0   lock = threading.RLock()   def Func():    lock.acquire()    global gl_num    gl_num +=1    time.sleep(1)    print gl_num    lock.release()       for i in range(10):    t = threading.Thread(target=Func)    t.start()

　　

Event

The events of the python thread are used by the main thread to control the execution of other threads. The events mainly provide three methods: set, wait, and clear.

Event processing mechanism: a global "Flag" is defined. If the "Flag" value is False, when the program executes the event. the wait method is blocked. If the "Flag" value is True, the event. the wait method is no longer blocked.

• Clear: Set "Flag" to False
• Set: set "Flag" to True

#!/usr/bin/env python# -*- coding:utf-8 -*- import threading  def do(event):    print 'start'    event.wait()    print 'execute'  event_obj = threading.Event()for i in range(10):    t = threading.Thread(target=do, args=(event_obj,))    t.start() event_obj.clear()inp = raw_input('input:')if inp == 'true':    event_obj.set()

Python Process

from multiprocessing import Processimport threadingimport time  def foo(i):    print 'say hi',i  for i in range(10):    p = Process(target=foo,args=(i,))    p.start()

　　

Note: because the data between processes must be one copy of each other, it is very costly to create a process.

Process data sharing

Each process holds one copy of data, and data cannot be shared by default.

Data cannot be shared between processes by default.

# Method 1: Arrayfrom multiprocessing import Process, Arraytemp = Array ('I', [,]) def Foo (I ): temp [I] = 100 + I for item in temp: print I, '----->', item for I in range (2): p = Process (target = Foo, args = (I,) p. start () # Method 2: manage. dict () shared data from multiprocessing import Process, Manager manage = Manager () dic = manage. dict () def Foo (I): dic [I] = 100 + I print dic. values () for I in range (2): p = Process (target = Foo, args = (I,) p. start () p. join ()

Type table

When a process is created (not used), the shared data is obtained to the sub-process. After the sub-process is executed, the original value is assigned.

Process lock instance Process pool

A process sequence is maintained in the Process pool. When used, a process is obtained from the process pool. If there is no usable process in the process pool sequence, the program will wait, until there are available processes in the process pool.

There are two methods in the process pool:

• Apply
• Apply_async

#! /Usr/bin/env python #-*-coding: UTF-8-*-from multiprocessing import Process, Poolimport time def Foo (I): time. sleep (2) return I + 100 def Bar (arg): print arg pool = Pool (5) # print pool. apply (Foo, (1,) # print pool. apply_async (func = Foo, args = (1 ,)). get () for I in range (10): pool. apply_async (func = Foo, args = (I,), callback = Bar) print 'end' pool. close () pool. join () # Shut down the processes in the process pool after they are executed. If comments are made, the program is closed directly.

Coroutine

The operations of threads and processes are system interfaces triggered by programs, and the executors are systems. The operations of coroutines are programmers.

The meaning of coroutine: For multi-threaded applications, the CPU uses slices to switch the execution between threads. It takes time to switch the threads (Save the status and continue next time ). Coroutine, only one thread is used to specify the execution sequence of a code block in one thread.

Application scenarios of coroutine: It is applicable to coroutine when there are a large number of operations without CPU (IO) in the program;

Greenlet

#!/usr/bin/env python# -*- coding:utf-8 -*-  from greenlet import greenlet  def test1():    print 12    gr2.switch()    print 34    gr2.switch()  def test2():    print 56    gr1.switch()    print 78 gr1 = greenlet(test1)gr2 = greenlet(test2)gr1.switch()

Gevent

import gevent def foo():    print('Running in foo')    gevent.sleep(0)    print('Explicit context switch to foo again') def bar():    print('Explicit context to bar')    gevent.sleep(0)    print('Implicit context switch back to bar') gevent.joinall([    gevent.spawn(foo),    gevent.spawn(bar),])

Automatic Switch upon IO operation:

from gevent import monkey; monkey.patch_all()import geventimport urllib2def f(url):    print('GET: %s' % url)    resp = urllib2.urlopen(url)    data = resp.read()    print('%d bytes received from %s.' % (len(data), url))gevent.joinall([        gevent.spawn(f, 'https://www.python.org/'),        gevent.spawn(f, 'https://www.yahoo.com/'),        gevent.spawn(f, 'https://github.com/'),])

Thread Pool:

 

Solution introduction:

 

Solution 1: Create a thread pool for a simple version every time;

Solution 2: Support for passing functions, passing parameters, returning call functions, and immediate termination of all threads. The biggest advantage is the loop utilization of threads, saving time and resources.★★★★★

Solution 3: The existing module can be called directly. Callback functions are not supported.

Solution 1:

#!/usr/bin/env python# -*- coding:utf-8 -*-import Queueimport threading  class ThreadPool(object):     def __init__(self, max_num=20):        self.queue = Queue.Queue(max_num)        for i in xrange(max_num):            self.queue.put(threading.Thread)     def get_thread(self):        return self.queue.get()     def add_thread(self):        self.queue.put(threading.Thread) """pool = ThreadPool(10) def func(arg, p):    print arg    import time    time.sleep(2)    p.add_thread()  for i in xrange(30):    thread = pool.get_thread()    t = thread(target=func, args=(i, pool))    t.start()"""

Solution 2:

#! /Usr/bin/env python #-*-coding: UTF-8-*-import queueimport threadingimport contextlibimport timeStopEvent = object () class ThreadPool (object ): def _ init _ (self, max_num): self. q = queue. queue () self. max_num = max_num self. terminal = False self. generate_list = [] self. free_list = [] def run (self, func, args, callback = None): "" the thread pool executes a task: param func: task function: param args: required parameter of the task function: param callback: task execution The callback function executed after a failure or success. The callback function has two parameters: 1. Execution status of the task function; 2. Return Value of the task function (default value: None, that is, no callback function is executed ): return: if the thread pool has been terminated, True is returned. Otherwise, None "" if len (self. free_list) = 0 and len (self. generate_list) <self. max_num: self. generate_thread () w = (func, args, callback,) self. q. put (w) def generate_thread (self): "Create a thread" "t = threading. thread (target = self. call) t. start () def call (self): "" loop to get the task function and execute the task function "current_thread = threading. curr EntThread self. generate_list.append (current_thread) event = self. q. get () while event! = StopEvent: func, arguments, callback = event try: result = func (* arguments) status = True failed t Exception as e: status = False result = e if callback is not None: try: callback (status, result) failed t Exception as e: pass if self. terminal: # False event = StopEvent else: with self. worker_state (self. free_list, current_thread): event = self. q. get () else: self. generate_list.remove (current_thread) @ contextlib. contextmanager def worker_state (self, x, v): x. append (v) try: yield finally: x. remove (v) def close (self): num = len (self. generate_list) while num: self. q. put (StopEvent) num-= 1 # terminate a thread (clear a queue) def terminate (self): self. terminal = True while self. generate_list: self. q. put (StopEvent) self. q. empty () import timedef work (I): time. sleep (1) print (I) pool = ThreadPool (10) for item in range (50): pool. run (func = work, args = (item,) # pool. terminate () # terminate all threads immediately

Solution 3,

From concurrent. futures import ThreadPoolExecutorimport timedef f1 (a): time. sleep (2) print (a) return 1 pool = ThreadPoolExecutor (5) for I in range (30): a = pool. submit (f1, I) # x =. result () # obtain the returned value. If yes, it will block