Python multi-thread and python multi-thread programming

Python multi-thread and python multi-thread programming

First, describe the multi-thread Application Scenario: When python processes multiple tasks, these tasks are essentially asynchronous and require multiple concurrent transactions, the running sequence of each transaction can be uncertain, random, and unpredictable. Computing-intensive tasks can be divided into multiple subtasks in sequence or processed in multiple threads. However, I/O-intensive tasks cannot be processed in a single thread mode. If multithreading is not required, only one or more timers can be used for implementation.

The process and thread are described as follows: a process (sometimes a heavyweight process) is an execution of a program, just as when we are in centos, ps-aux | grep something, there is always a process generated by itself, that is, this grep process. Each process has its own address space, memory, data stack, and other auxiliary data that records its running trajectory, therefore, each process cannot directly share information and can only use inter-process communication (IPC ).

The biggest difference between a thread (Lightweight Process) and a process is that all threads run in the same process, share the same runtime environment, and share the same piece of data space. Therefore, threads can share data and communicate with each other more conveniently than processes, and execute transactions concurrently.

To facilitate understanding of the relationship between memory processes and threads, we can make an analogy: Compare cpu to a moving company, and this moving company only has one car (process) for use, this moving company is very poor and has only one employee (single thread). So, this moving company can only move up to five companies a day. Later, the boss made money and he didn't buy a car, instead, we hired n more employees (multi-thread). In this way, each employee will be arranged to move only one employee each time, and then they will go to the rest, get the car out, and let others move the next one, this does not seem to improve much efficiency, but increases costs, right? This is because GIL (Global Interpreter Lock) Global Interpreter Lock, this ensures thread security (Secure Data Reading), that is, only one thread can run on the CPU. This is a unique mechanism of python, that is, even if your running environment has dual CPU, the python virtual machine also uses only one cpu, that is, GIL directly causes CPython to not use the performance of multiple physical cores to accelerate computation. For detailed explanations (problems left over by history, hardware is growing too fast), refer to this blog:

Http://blog.sina.com.cn/s/blog_64ecfc2f0102uzzf.html

In core python programming, the author strongly recommends that we use the threading module instead of the thread module. The reason is as follows:

1. When the main thread exits, all other threads are exited without being cleared. The thread module cannot protect the safe exit of all sub-threads. That is, the thread module does not support daemon.

2. The attributes of the thread module may conflict with threading.

3. There are few synchronization primitives for low-level thread modules (actually only one, should be sleep ).

I. thread Module

The following two sample codes do not use GIL or use GIL:

1. Example code without GIL:

1 from time import sleep, ctime 2 import thread 3 4 def loop0 (): 5 print 'start loop 0 at: ', ctime () 6 sleep (4) 7 print 'loop 0 done at: ', ctime () 8 def loop1 (): 9 print 'start loop 1 at:', ctime () 10 sleep (2) 11 print 'loop 1 done at: ', ctime () 12 def main (): 13 print 'start at:', ctime () 14 thread. start_new_thread (loop0, () 15 thread. start_new_thread (loop1, () 16 sleep (6) 17 print 'all loop is done, ', ctime () 18 19 if _ name __= = '_ main _': 20 main () 21 22 23 output result: 24 25 start: thu Jan 28 10:46:27 201626 start loop 0 at: Thu Jan 28 10:46:27 201627 28 start loop 1 at: Thu Jan 28 10:46:27 201629 loop 1 done: thu Jan 28 10:46:29 201630 loop 0 done at: Thu Jan 28 10:46:31 201631 all loop is done, Thu Jan 28 10:46:33 2016

From the above output, we can see that we have successfully enabled two threads and synchronized them with the main thread. In the second s, loop1 is completed first, loop0 is completed at 4s, and 2 s is passed, the main thread is finished. The entire main thread goes through 6 s, and loop0 and loop1 are synchronized.

 

2. sample code using GIL:

 

1 import thread 2 from time import sleep, ctime 3 loops = [4, 2] 4 def loop (nloop, nsec, lock): 5 print 'start loop ', nloop, 'at: ', ctime () 6 sleep (nsec) 7 print 'loop', nloop, 'done at:', ctime () 8 lock. release () 9 def main (): 10 print 'starting at: ', ctime () 11 locks = [] 12 nloops = range (len (loops )) 13 14 for I in nloops: 15 lock = thread. allocate_lock () # The list of locks to be created, which contains 16 locks in locks. acquire () 17 locks. append (lock) 18 for I in nloops: 19 thread. start_new_thread (loop, (I, loops [I], locks [I]) # create a thread. The parameter is the loop number, sleep time, and the lock is 20 for I in nloops: 21 while locks [I]. locked (): # Wait until the loop is completed. Unlock 22 pass23 print 'all DONE at: ', ctime () 24 if _ name _ =' _ main __': 25 main () 26 27 28 and above output: 29 30 starting at: Thu Jan 28 14:59:22 201631 start loop 0 at: Thu Jan 28 14:59:22 201632 33 start loop 1: thu Jan 28 14:59:22 201634 loop 1 done at: Thu Jan 28 14:59:24 201635 loop 0 done at: Thu Jan 28 14:59:26 201636 all DONE at: Thu Jan 28 14:59:26 2016

 

It takes 4 seconds, which improves the efficiency and is more reasonable than using a sleep () function in the main thread.

 

Ii. threading Module

1. Thread class

In the thread class, you can use the following three methods to create a thread:

(1) create a thread instance and pass it a function

(2) create a thread instance and pass it a callable Class Object

(3) derive a subclass from thread and create the object of this subclass.

Method (1)

1 _ author _ = 'Dell '2 import threading 3 from time import sleep, ctime 4 def loop0 (): 5 print 'start loop 0 at: ', ctime () 6 sleep (4) 7 print 'loop 0 done at: ', ctime () 8 def loop1 (): 9 print 'start loop 1 at:', ctime () 10 sleep (2) 11 print 'loop 1 done at: ', ctime () 12 def main (): 13 print 'starting at:', ctime () 14 threads = [] 15 t1 = threading. thread (target = loop0, args = () # create Thread 16 threads. append (t1) 17 t2 = threading. thread (target = loop1, args = () 18 threads. append (t2) 19 for t in threads: 20 t. setDaemon (True) <span style = "white-space: pre"> </span> # enable the daemon thread (must be called before start () 21 t. start () <span style = "white-space: pre"> </span> # start thread execution 22 for t in threads: <span style = "white-space: pre "> </span> 23 TB. join () <span style = "white-space: pre"> </span> # suspends the program until the thread ends. If a value is given, A maximum of 24 25 if _ name _ = '_ main _': 26 main () 27 print 'all DONE at: ', ctime () 28 29 here, we don't need to manage so many locks (allocation, acquisition, release, check, etc.) as the thread module does. At the same time, I also reduced the circular code, the result is as follows: 30 31 32 starting at: Thu Jan 28 16:38:14 201633 start loop 0 at: Thu Jan 28 16:38:14 201634 start loop 1: thu Jan 28 16:38:14 201635 loop 1 done at: Thu Jan 28 16:38:16 201636 loop 0 done at: Thu Jan 28 16:38:18 201637 All DONE at: Thu Jan 28 16:38:18 2016

The results are the same, but the logic of the Code is much clearer. The other two types will not post the code here. Instantiate a Thread and call a thread. the biggest difference between start_new_thread and start_new_thread is that the new Thread will not be executed immediately. That is to say, the Thread class of the threading module will be called after it is instantiated. the start () function is executed in a unified manner, which makes our program have a good synchronization feature.

The following is a comparison between a single thread and a multi-thread. The two groups of operations are completed by multiplication and division respectively, so that the efficiency of multiple threads is improved.

1 from time import ctime, sleep 2 import threading 3 4 def multi (): 5 num1 = 1 6 print 'start mutiple at: ', ctime () 7 for I in range (0.2): 8 num1 = I * num1 9 sleep () 10 print 'mutiple finished at: ', ctime () 11 return num112 def divide (): 13 num2 = 10014 print 'start division at: ', ctime () 15 for I in range (0.4): 16 num2 = num2/i17 sleep) 18 print 'division finished at: ', ctime () 19 return num220 def main (): 21 print' ----> single Thread '22 x1 = multi () 23x2 = divide () 24 print 'the sum is ', sum ([x1, x2]),' \ nfinished singe thread', ctime () 25 26 print '-----> Multi Thread '27 threads = [] 28 t1 = threading. thread (target = multi, args = () 29 threads. append (t1) 30 t2 = threading. thread (target = divide, args = () 31 threads. append (t2) 32 for t in threads: 33 t. setDaemon (True) 34 t. start () 35 for t in threads: 36 t. join () 37 38 if _ name _ = '_ main _': 39 main () 40 41 results: 42 43 44 45 ----> single Thread46 47 start mutiple at: Thu Jan 28 21:41:18 201648 49 mutiple finished at: Thu Jan 28 21:41:20 201650 51 start division: thu Jan 28 21:41:20 201652 53 division finished at: Thu Jan 28 21:41:24 201654 55 The sum is 362880 56 57 finished singe thread Thu Jan 28 21:41:24 201658 59 -----> Multi Thread60 61 start mutiple: thu Jan 28 21:41:24 201662 63 start division at: Thu Jan 28 21:41:24 201664 65 mutiple finished at: Thu Jan 28 21:41:26 201666 67 division finished at: Thu Jan 28 21:41:27 201668 69 The sum is: 362880