Python multi-process multi-line programming templates and python multi-line programming templates

Python multi-process multi-line programming templates and python multi-line programming templates

I. Introduction to the principle of multi-process and multi-thread in Python

1. Python global interpreter lock GIL

A) The GIL global interpreter lock of Python is a mutex lock, which can prevent multiple local threads from executing Python bytecode at a time; CPython memory management is not secure (Memory leakage) at the Thread level, so this global interpreter lock is required. Each Python process can only apply to use one GIL lock. Therefore, Python multithreading is concurrent but cannot be processed in parallel. The interpreter of Python can only execute one thread at a time. After the GIL lock is released, the next thread is executed in turn.

B) In Python2.x, the GIL release logic is that the current thread encounters an IO operation or the ticks count reaches 100. In python3.x, GIL does not use ticks count, instead, use a timer to release the GIL of the current thread after the execution time reaches the set threshold.

2. Python multi-process and multi-thread application scenarios

A) CPU-bound (computing-intensive)

CPU-bound indicates that the system's hard disk/memory performance is much better than the CPU performance. The status of the system during operation is that the CPU usage is close to 100%, and the I/O read/write hard disk and memory usage is very low. In this computing-intensive situation, when Python's ticks counter quickly reaches the threshold, GIL is released and relocked. At this time, multiple threads in Python will frequently lock and release locks, consuming a lot of CPU resources. Therefore, it is much better for computing-intensive programs to use multiple processes in Python than multithreading.

B) I/O-bound (I/O-intensive)

I/O-bound indicates that the CPU performance of the system is much better than that of the hard disk/memory. Most of the time the system runs is waiting for I/O read/write, and the CPU usage is not high. When I/O operations are performed in A single Python thread, I/O waits, causing unnecessary CPU time waste, automatically switch to thread B, which can improve program execution efficiency without wasting CPU resources. Therefore, I/O-intensive program Python multithreading has some advantages.

Ii. Python multi-process multi-thread programming Template

1. This template has two important advantages:

A) customize the number of multi-process and multi-thread enable based on the Program Execution type to adjust the program execution efficiency.

B) The execution progress of the program is also printed in real time, which does not significantly affect the execution efficiency of the program.

 

1 #! /Usr/bin/env python 2 #-*-coding: UTF-8-*-3 _ author __= "A Hui Fengqing" 4 _ date _ = "$21:49:51 $" 5 import OS 6 import sys 7 import time 8 import random 9 import threading 10 from multiprocessing import Process, manager, Lock 11 12 # fixed encoding Issue 13 reload (sys) 14 sys. setdefaultencoding ('utf-8') 15 Type = sys. getfilesystemencoding () 16 17 # ------------------------------------------------ 18 # modifiable Global Changes Parameter -- Start. 19 tasklist = [] # define the task list 20 21 PROCESS_COUNT = 2 # define the number of processes 22 THREAD_COUNT = 4 # define the number of threads 23 # modify the global variable parameter -- End. 24 # ---------------------------------------------- 25 26 27 class HandleTask (threading. thread): 28 "docstring for HandleTask" 29 30 def _ init _ (self, proid, prolock, thrid, thrlock, tasklist, tasknum, schedule ): 31 super (HandleTask, self ). _ init _ () 32 self. proid = proid 33 self. prolock = prolock 34 self. thrid = thrid 35 self. thrlock = thrlock 36 self. tasklist = tasklist 37 self. tasknum = tasknum 38 self. sch = schedule 39 self. pid = OS. getpid () 40 41 def run (self): 42 self. prolock. acquire () 43 self. thrlock. acquire () 44 print "The Thread [% s: % s] tasklist number: [% s]" % (self. proid, self. thrid, len (self. tasklist) 45 self. thrlock. release () 46 self. prolock. release () 47 48 for (element,) in self. tasklist: 49 # task execution starts 50 # print element 51 time. sleep (1) 52 # task execution ended 53 54 self. prolock. acquire () 55 self. thrlock. acquire () 56 self. sch. value + = 1 57 self. thrlock. release () 58 self. prolock. release () 59 60 def Thread_Handle (proid, prolock, tasklist, tasknum, schedule): 61 global THREAD_COUNT 62 lock = threading. lock () 63 WorksThread = [] 64 thread_task_number = len (ta Sklist)/THREAD_COUNT 65 if thread_task_number = 0: 66 THREAD_COUNT = len (tasklist) 67 thread_task_number = 1 68 69 for I in range (THREAD_COUNT): 70 if I! = THREAD_COUNT-1: 71 source_list = tasklist [I * thread_task_number: (I + 1) * thread_task_number] 72 else: 73 source_list = tasklist [I * thread_task_number:] 74 Work = HandleTask (proid, prolock, I, lock, source_list, tasknum, schedule) 75 Work. start () 76 WorksThread. append (Work) 77 78 for Work in WorksThread: 79 Work. join () 80 81 def Process_Handle (tasklist, tasknum): 82 global PROCESS_COUNT 83 l Ock = Lock () 84 # define the progress variable schedule 85 schedule = Manager (). value ('schedule', 0) 86 WorksProcess = [] 87 # Process Task Allocation by task size 88 process_task_num = len (tasklist)/PROCESS_COUNT 89 if process_task_num = 0: 90 PROCESS_COUNT = len (tasklist) 91 process_task_num = 1 92 93 for I in range (PROCESS_COUNT): 94 if I! = PROCESS_COUNT-1: 95 source_list = tasklist [I * process_task_num: (I + 1) * process_task_num] 96 else: 97 source_list = tasklist [I * process_task_num:] 98 Work = Process (target = Thread_Handle, args = (I, lock, source_list, tasknum, schedule) 99 Work. start () 100 WorksProcess. append (Work) 101 # add an additional Process to print the task execution progress 102 Work = Process (target = Displays, args = (lock, tasknum, schedule) 103 Work. start () 104 WorksPr Ocess. append (Work) 105 for Work in WorksProcess: 106 Work. join () 107 del WorksProcess108 109 def Displays (prolock, tasknum, schedule, delaytime = None): 110 if delaytime is None: 111 delaytime = 1112 while (tasknum-schedule. value): 113 time. sleep (delaytime) 114 print "Completed: [% s], Remaining: [% s]" % (schedule. value, tasknum-schedule. value) 115 116 def main (): 117 # print The output master Process number 118 print "The Main Process ID: [% s] "% OS. getpid () 119 # create a test task 120 for I in range (1,101): 121 tasklist. append (I,) 122 Process_Handle (tasklist, len (tasklist) 123 124 125 if _ name _ = '_ main __': 126 127 print "The Program start time:", time. strftime ("% Y-% m-% d % H: % M: % S", time. localtime () 128 start = time. time () 129 main () 130 print "The Program end time:", time. strftime ("% Y-% m-% d % H: % M: % S", time. localtime (), "[% s]" % (time. tim E ()-start) 131 raw_input ("Please enter any key to end !!! ". Decode ('utf-8'). encode (Type ))

View Code

2. Program Execution test results

 1 The Program start time: 2017-05-12 18:15:12 2 The Main Process ID:[9752] 3 The Thread [0:0] tasklist number:[12] 4 The Thread [1:0] tasklist number:[12] 5 The Thread [1:1] tasklist number:[12] 6 The Thread [0:1] tasklist number:[12] 7 The Thread [1:2] tasklist number:[12] 8 The Thread [0:2] tasklist number:[12] 9 The Thread [0:3] tasklist number:[14]10 The Thread [1:3] tasklist number:[14]11 Completed:[0] , Remaining:[100]12 Completed:[8] , Remaining:[92]13 Completed:[16] , Remaining:[84]14 Completed:[25] , Remaining:[75]15 Completed:[34] , Remaining:[66]16 Completed:[42] , Remaining:[57]17 Completed:[51] , Remaining:[49]18 Completed:[59] , Remaining:[41]19 Completed:[67] , Remaining:[32]20 Completed:[76] , Remaining:[24]21 Completed:[85] , Remaining:[15]22 Completed:[93] , Remaining:[7]23 Completed:[98] , Remaining:[2]24 Completed:[100] , Remaining:[0]25 The Program end time: 2017-05-12 18:15:27 [15.007999897]26 Please enter any key to end!!!

View Code