Python --- process and thread, python --- process thread
Processes and threads what are threads and processes
Thread: The minimum unit that the operating system can schedule operations. It is included in the process and is the actual operating unit of the process. Is a collection of commands.
A thread refers to a single sequential control flow in a process. A process can be concurrently executed by multiple threads, and each thread can execute different tasks in parallel.
Process: The process is exposed to the operating system management in the form of a whole, including the call to various resources, memory management, network interface calls, and a set of various resource management, it can be called a process.
To operate the cpu, a thread must be created first. All threads in the same process share the same memory space.
Note: 1. The process itself cannot be executed
2. Process and thread cannot compare who is fast and who is slow. There is no comparability between the two. Process is a collection of resources, the thread is actually executing the task, and the process needs to run the task through the thread
3. Start a thread faster than start a process
Differences between processes and threads
1. The thread shares the memory space, and the memory of the process is independent.
2. thread have direct access to data segment of its process; process have their own copy of the data segment of the parent process
3. threads of the same process can communicate with each other. To communicate with each other, you must use an intermediate proxy.
4. It is easy to create a thread. To create a new process, you need to clone its parent process.
5. One thread can control and operate other threads in the same process, but the process can only operate sub-processes.
6. Changing a main thread may affect the running of other threads. Modifying the parent process will not affect the child process.
Python GIL (Global interpreter lock)
In python, no matter how many threads you enable and how many CPUs you have, only one thread is allowed to run at the same time during python execution.
Cause: the python thread is the native thread that calls the operating system.
Thread lock:
1 import threading 2 import time 3 def run (n): 4 lock. acquire () 5 global num 6 num + = 1 7 # time. sleep (1) do not use sleep when locking; otherwise, wait 50 s to complete 8 locks. release () 9 lock = threading. lock () 10 num = 011 start_time = time. time () 12 t_objs = [] 13 for I in range (50): 14 t = threading. thread (target = run, args = ("t-% s" % I,) 15 t. start () 16 t_objs.append (t) 17 for t in t_objs: 18 t. join () 19 print ("----- all threads has finished .... ", threading. current_thread (), threading. active_count () 20 21 print ("num =", num)
Locks the thread and turns the program into a serial number to ensure that num is accurate when + 1 is used. Otherwise, a thread is executing + 1 and has not ended yet, another thread starts to + 1, and the final result cannot be accurate.
Recursive lock: contains a small lock in a large lock
Scenario: after school, the student leaves school and suddenly finds that his pencil case falls into the teacher. The school only allows one person to enter the school at a time (this person is responsible for the loss of other things ), the classroom door should also be opened when you enter the classroom (to prevent others from leaving the school), and finally get the pencil case and leave the school. During this period, the school door is equivalent to a lock, and the classroom door is equivalent to another lock. This ensures no interference from others throughout the process.
1 import threading,time 2 3 def run1(): 4 print("grab the first part data") 5 lock.acquire() 6 global num 7 num += 1 8 lock.release() 9 return num10 11 def run2():12 print("grab the second part data")13 lock.acquire()14 global num215 num2 += 116 lock.release()17 return num218 19 def run3():20 lock.acquire()21 res = run1()22 print('------between run1 and run2')23 res2 = run2()24 lock.release()25 print(res,res2)26 27 num,num2 = 0,028 lock = threading.RLock()29 for i in range(10):30 t = threading.Thread(target=run3)31 t.start()32 33 while threading.active_count() != 1:34 print(threading.active_count())35 else:36 print("----all threads done---")37 print(num,num2)
It is equivalent to first entering the run3 gate, then entering the run1 and run2 respectively, and then adding 1 to num1 and num2 to ensure that the execution of each thread is sequential.
Python threading Module
There are two calling methods in the thread
Direct call:
1 import threading 2 import time 3 4 def run(n): 5 print("task ",n) 6 time.sleep(2) 7 8 t1 = threading.Thread(target = run,args = ("t1",)) 9 t2 = threading.Thread(target = run,args = ("t2",))10 11 t1.start()12 t2.start()
Running result: two results are generated at a time, but the program will wait for 2 seconds to end. Two seconds in total, because they are in parallel.
Inherited call:
1 import threading 2 import time 3 4 # Start thread 5 class MyThread (threading. thread): 6 def _ init _ (self, n): 7 super (MyThread, self ). _ init _ () 8 self. n = n 9 def run (self): 10 print ("running task", self. n) 11 12 t1 = MyThread ("t1") 13 t2 = MyThread ("t2") 14 t1.start () 15 t2.start ()
Only two threads are started, and 50 threads are started below:
1 import threading 2 import time 3 4 def run(n): 5 print("task ",n) 6 time.sleep(2) 7 8 start_time = time.time() 9 for i in range(50):10 11 t = threading.Thread(target = run,args = ("t-%s" %i,))12 t.start()13 print("cost : ",time.time()-start_time)
Running result:
Problem: it only takes this time after 50 processes are created. The main program does not wait for other child threads to go down.
Cause: this is the reason for multithreading. A program has at least one thread. The program itself is a thread. The main thread. After the main thread starts the sub-thread, the sub-thread has no relationship with the main thread, two do not affect each other
Solution: join () method. The main program goes down after the sub-thread completes running.
import threadingimport timedef run(n): print("task ",n) time.sleep(2) print("task has done.... ",n)start_time = time.time()t_objs = []for i in range(50): t = threading.Thread(target = run,args = ("t-%s" %i,)) t.start() t_objs.append(t)for t in t_objs: t.join()print("------all threads has finished")print("cost : ",time.time()-start_time)
Create a temporary list, store the created threads, and use the join () method one by one.
50 threads are created, and the main thread is the program itself. That is to say, there are 51 threads in total.
# Threading. current_thread () shows whether the current thread is a main thread or a subthread # threading. active_count () displays the number of threads currently running print ("----- all threads has finished .... ", threading. current_thread (), threading. active_count ())Daemon
T. setDaemon (True) # set the current thread to a daemon thread before start ().
When the non-daemon thread ends, all daemon threads exit.
1 import threading 2 import time 3 def run (n): 4 print ("task", n) 5 time. sleep (2) 6 print ("task has done .... ", n) 7 start_time = time. time () 8 t_objs = [] 9 for I in range (50): 10 11 t = threading. thread (target = run, args = ("t-% s" % I,) 12 t. setDaemon (True) # set the current thread to a daemon thread, 13 TB before start. start () 14 t_objs.append (t) 15 16 print ("----- all threads has finished .... ", threading. current_thread (), threading. active_count () 17 print ("cost:", time. time ()-start_time)
Running result: the task has done is not running. After the main thread ends, all child threads end forcibly.
Semaphore (Semaphore)
The mutex lock allows only one thread to change data at the same time, while Semaphore allows a certain number of threads to change data at the same time. If there are three seats, a maximum of three persons are allowed to change data at the same time, the people behind the scenes can sit in only when someone gets up.
1 import threading, time 2 def run (n): 3 semaphore. acquire () 4 time. sleep (1) 5 print ("run the thread: % s \ n" % n) 6 semaphore. release () 7 8 if _ name _ = '_ main _': 9 10 semaphore = threading. boundedSemaphore (5) # A maximum of five threads can be run simultaneously. 11 for I in range (20): 12 t = threading. thread (target = run, args = (I,) 13 t. start () 14 15 16 while threading. active_count ()! = Pass # print (threading. active_count () 18 else: 19 print ("---- all threads done ---") 20 21 # Five threads are executed at a time.
Similar to thread locks, both acquire () and release () are required. The difference is that there are multiple locks in semaphores,
Threading. boundedSemaphore (5) allows a maximum of five threads to run at the same time, and the running result is five results together, however, in essence, if three of the five are completed first, they will be immediately sent to three. It will not wait until five are completed. It will be put into one for each of them.
If the five data changes at the same time, the error may be corrected. This is mainly used for: link pool and thread pool. Mysql connections have the concept of a link pool, with a maximum of several concurrent connections at the same time. For socketserver, to ensure that the system is not slow by too many threads, you can use semaphores to obtain a maximum of 100 connections at the same time point.
Event)
a events is a simple synchronization object;
the event represents an internal flag,and threads can wait for the flag to be set,or set or clear the flag themselves
An event is a simple synchronization object. An event is equivalent to setting all the variables. Then, the thread continuously detects the changes of this variable and then performs different operations.
Method:
Event = threading. Event () generates an event object
Event. set ()
Event. clear ()
Event. wait ()
If the flag is set, the wait method doesn't do anything
The flag is set to indicate a green light and direct access.
If the flag is cleared, wait will block until it becomes set again
The flag is cleared, indicating a red light. wait waits for a green light.
Any number of threads my wait for the same event
Traffic light case:
1 import threading 2 import time 3 4 event = threading. event () 5 def lighter (): 6 count = 0 7 event. set () # set green light to 8 at first while True: 9 if count> 5 and count <= 10: # change to red light 10 event. clear () # clear flag 11 print ("\ 033 [41; 1 mred light is on... \ 033 [0 m ") 12 elif count> 10: 13 event. set () # change the green light 14 print ("\ 033 [42; 1 mgreen light is on... \ 033 [0 m ") 15 count = 016 else: 17 print (" \ 033 [42; 1 mgreen light is on... \ 033 [0 m ") 18 time. sleep (1) 19 count ++ = 120 def car (name): 21 while True: 22 if event. is_set (): # indicates the green light 23 print ("[% s] running... "% name) 24 time. sleep (1) 25 else: 26 print ("[% s] sees red light, waiting... "% name) 27 event. wait () 28 print ("\ 033 [34; 1 m [% s] green light is on, start going... \ 033 [0 m "% name) 29 30 light = threading. thread (target = lighter,) 31 light. start () 32 car1 = threading. thread (target = car, args = ("Tesla",) 33 car1.start ()
Running result:
Queue
Queue is especially useful in threaded programming when information must be exchanged safely between multiple threads
Class queue. Queue (maxsize = 0) first in first out
Class queue. LifeQuene (maxsize = 0) last in first out
Class queue. PriorityQueue (maxsize = 0) a queue with a priority can be set during data storage.
A queue is an ordered container.
What is the difference between a list and a queue? After a list is retrieved, the data is still in the list, such as copying data. There is only one copy of the data in the queue.
Class queue. Queue:
Use the put () and get () Methods to add or subtract data from the queue and use the qsize () method to obtain the amount of data in the queue at the moment.
But there is a problem: when the data is obtained, if the get () method is used to retrieve the data, the program will
You can use the get_nowait () method. If there is no data in the queue, an exception will be thrown, so that it will not be stuck. You can also use the qsize () method to determine whether to use it, if no data exists, do not retrieve it.
Queue. get (block = True, timeout = None) the block parameter. If no data is obtained, it will be stuck by default. If it is set to false, it will not be stuck. The timeout setting will be stuck for several seconds.
Class queue. LifoQueue (maxsize = 0) # last in first out
1 import queue2 q = queue.LifoQueue()3 q.put(1)4 q.put(2)5 q.put(3)6 print(q.get())7 print(q.get())8 print(q.get())
Running result:
Class queue. PriorityQueue (maxsize = 0) # You can set a priority queue in the data storage room.
1 import queue 2 q = queue.PriorityQueue() 3 4 q.put((10,"d1")) 5 q.put((-1,"d2")) 6 q.put((3,"d4")) 7 q.put((6,"d5")) 8 print(q.get()) 9 print(q.get())10 print(q.get())11 print(q.get())
Running result:
Producer consumer model:
In concurrent programming, the use of producer and consumer modes can solve most concurrency problems. This mode balances the working capabilities of production lines and consumption threads to increase the overall data processing speed of the program.
Example:
1 import threading, time 2 import queue 3 q = queue. queue (maxsize = 10) 4 def Producer (name): 5 count = 1 6 while True: 7 q. put ("bone: % s" % count) 8 print ("produced bone:", count) 9 count + = 110 time. sleep (0.5) 11 12 13 14 def Consumer (name): 15 while True: 16 if q. qsize ()> 0: 17 print ("[% s] GET [% s] and eat it... "% (Name, q. get () 18 time. sleep (1) 19 p = threading. thread (target = Producer, args = ("xiaoming",) 20 c = threading. thread (target = Consumer, args = ("xiaohong",) 21 c1 = threading. thread (target = Consumer, args = ("liangliang",) 22 p. start () 23 c. start () 24 c1.start ()
Running result: