Python Concurrency and threading

Concurrency and threading Basic concepts-parallel, concurrency

parallel , parallel

Do multiple things at the same time with no interference;

For example, at the same time, there are multiple vehicles running on multiple lanes, the concept of simultaneous occurrence.

concurrency , concurrency

Do something at the same time, but emphasize doing more things at the same time.

For example, at the same intersection, a vehicle has to pass the road at the same time as the incident.

queues, buffers

Similar queueing is a natural solution to concurrency. The queued area is the buffer.

Resolving concurrency :

"Canteen model", 12 o'clock Noon, everyone rushed to the canteen, is concurrent. A lot of people are high concurrency. "

1, queue, buffer:

Queue: That is, queued.

Buffers: Queues that are queued.

Priority queue: If there is a team of boys and girls, the girls ' team priority is the priority queue.

2. Scramble for:

Lock mechanism: Fight for a meal, someone grabbed, the window in a moment can only serve this person, lock window, that is, lock mechanism.

Scrambling is also a high-concurrency solution, but it is not recommended to have someone who can not get it for a long time.

3. Pretreatment:

Statistics everyone loves to eat food, favorite 80% hot dishes in advance, 20% unpopular dishes are now done, so even if someone locked the window, can be released quickly.

This is a kind of pre-loading the user needs data ideas, preprocessing ideas, cache commonly used.

4. Parallel:

Open more than one meal window and provide service at the same time.

It can be processed in parallel by purchasing more servers, or multiple threads, to solve concurrency problems.

This is an idea of horizontal expansion.

Note: If a thread is processed on a single CPU, it is not parallel.

5, speed up:

By increasing the speed of a single window, it is also a way to solve concurrency.

It improves individual CPU performance, or a single server installs more CPUs.

This is an idea of vertical expansion.

6. Message Middleware:

such as the nine ileum of the subway station in the corridor, cushion the flow of people.

Common message middleware: RabbitMQ, ActiveMQ (Apache), ROCKETMQ (Ali Apache), Kafka (Apache), etc.

The above example shows that technology comes from life! 】

Processes and Threads

A) in the operating system that implements the thread, the thread is the smallest unit in which the operating system can operate the dispatch.

b) The thread is included in the process and is the actual operating unit of the process.

c) The execution instance of a program is a process.

Process is a computer program on a data set on a running activity, the system is the basic unit of resource allocation and scheduling, is the basis of the operating system structure.

Process and program relationships: a process is a container for threads.

The Linux process is divided between the parent and child processes, and the Windows process is equal.

Threads are sometimes called lightweight processes, and a standard thread consists of thread IDs, current instruction pointers, register collections, and stacks.

When you run a program, the OS creates a process. It uses system resources (CPU, memory, and disk space) and data structures in the OS kernel (files, network Connections, usage statistics, and so on).

Processes are isolated from one another, that is, a process cannot access the contents of other processes or manipulate other processes.

The operating system keeps track of all running processes, gives each process a small run time, and then switches to other processes, which can be both fair and responsive to user actions.

You can view the status of a process in the graphical interface, and you can use Task Manager on Windows. You can also write your own program to get process information.

# Gets the process number and current working directory of the running Python interpreter, as well as the user ID, user group ID.

In [1]: Import Osin [2]: Os.getpid () out[2]: 2550In [3]: Os.getuid () out[3]: 0 in [4]: OS.GETCWD () out[4]: '/root ' in [5]: OS . Getgid () Out[5]: 0 in [6]:

　　

Understanding Threads, Threads:

• The process is an independent kingdom, and it is not possible to share data freely among processes.
• Threads are provinces, and threads within the same process can share the resources of a process, each of which has its own independent stack.

The state of the thread:

• Ready: Once the thread is running, it is waiting to be dispatched.
• Run (Running): Thread is running.
• Blocking (Blocked): Threads wait for external events to occur and cannot run, such as I/O operations.
• Terminate (Terminated): Thread completes or exits, or is canceled.

Processes and threads in Python: The process initiates an interpreter process, and the thread shares an interpreter process.

Thread development for Python

Python thread development uses standard library threading.

Thread class

# signature

def __init__ (self, group=none, Target=none, Name=none, args= (), Kwargs=none, *, Daemon=none)

　

• Target: The object called by the thread, which is the objective function.
• Name: a name for the thread.
• Args: Pass arguments, tuples, for the target function.
• Kwargs: The dictionary that is the keyword for the target function.

Thread Start

Import threading

# The simplest threaded thread

def worker ():    print ("I ' m working")    print ("finished") T = Threading. Thread (target=worker, name= ' worker ')  # Threading object. T.start ()

through threading. thread creates a threaded object , target is the destination function, and name can specify a name.

However, the thread does not start , and the start method needs to be called .

A thread executes a function because the thread is executing code, and the simplest package is a function, so it is called a function.

After the function executes, the thread exits as well.

If you do not let the thread exit, or let the thread work: use a while loop inside the function.

Import Threadingimport timedef worker (): While    True:        time.sleep (1)        print ("I ' m work")    print (' Finished ') T = Threading. Thread (target=worker, name= ' worker ')  # Threading object. T.start ()   # start.

　　

Thread exits

Python does not provide a way to exit the thread, and exits when the following conditions occur:

• The execution of the statement inside the thread function is complete.
• An unhandled exception was thrown in the thread function.

Import threadingimport time   def worker ():    count = 0 while    True:        if (Count > 5):            Raise RuntimeError ()            # return        time.sleep (1)        print ("I ' m working")        count + = 1t = Threading. Thread (target=worker, name= ' worker ')  # Threading object. T.start ()  # start. Print ("==end==")

python thread does not have a priority, There is no concept of thread groups, nor can it be destroyed, stopped, suspended, naturally, there is no recovery or interruption.

Thread's pass-through parameter

Import threadingimport Time def add (x, y):    print (' {} + {} = {} '. format (x, y, x + y, Threading.current_thread ())) Threa D1 = Threading. Thread (Target=add, name= ' Add ', args= (4, 5))  # Threading object. Thread1.start ()  # start. Time.sleep (2) thread2 = Threading. Thread (Target=add, name= ' Add ', args= (5,), kwargs={' Y ': 4})  # Threading object. Thread2.start ()  # start. Time.sleep (2) Thread3 = Threading. Thread (Target=add, name= ' Add ', kwargs={' x ': 4, ' Y ': 5})  # Threading object. Thread3.start ()  # starts.

　　

There is no difference between a line Cheng and a function parameter, which is essentially a function pass.

Properties and methods of threading

Current_thread () # Returns the current thread object.

Main_thread () # Returns the main thread object.

Active_count () # The number of threads currently in the Alive State.

Enumerate () # Returns a list of all live threads, excluding threads that have been terminated and threads that have not started.

Get_ident () # returns the current thread ID, not 0 integers.

The value returned by the Active_count, enumerate method also includes the main thread.

Import threadingimport Time Def showthreadinfo ():    print (' CurrentThread = {} '. Format (Threading.current_thread ()))    print (' main thread = {} '. Format (Threading.main_thread ()), ' main thread object ')    print (' Active count = {} '. Format ( Threading.active_count ()), ' alive ') def worker ():    count = 1    showthreadinfo () while    True:        if (count > 5):            break        time.sleep (1)        count + = 1        print ("I ' m working") T = Threading. Thread (target=worker, name= ' worker ')  # Threading object. Showthreadinfo () T.start ()  # start. Print (' ==end== ')

　　

Properties and methods for thread instances

Name: It is just a name identifier that can be named, GetName (), SetName () to get, set this noun.

Ident: Thread ID, which is a non-0 integer. The thread does not have an ID until it is started, otherwise none. Thread exits, this ID is still accessible. This ID can be reused.

Is_alive (): Returns whether the thread is alive.

Note: The thread name is a name that can be duplicated; The ID must be unique, but can be used after the thread exits.

Import Threadingimport Time def worker ():    count = 0 while    True:        if (Count > 5):            Break        Time.sleep ( 1)        count + = 1        print (Threading.current_thread (). Name, ' ~~~~~~~~~~~~~~~~~~~~~~~ ') T = Threading. Thread (name= ' worker ', target=worker) print (t.ident) T.start () while True:    time.sleep (1)    if T.is_alive ():        print (' {} {} alive '. Format (T.name, t.ident))    else:        print (' {} {} dead '. Format (T.name, t.ident)) T.start ()

　　

Start (): Starts the thread. Each thread must and can only execute the method one time.

Run (): Runs the thread function.

To demonstrate, derive a thread subclass

# Start method.

Import Threadingimport Time def worker ():    count = 0 while    True:        if (Count >= 5):            break        Time.sleep (1)        Count + = 1        print (' Worker running ') class MyThread (threading. Thread):    def start:        print (' start~~~~~~~~~~~~~ ')        super (). Start ()     def run (self):        print (' run~~~~~~~~~~~~~~~~~ ')        Super (). Run () t = MyThread (name= ' worker ', Target=worker) T.start ()

Run method

Import Threadingimport Time def worker ():    count = 0 while    True:        if (Count > 5):        break Time.sleep (1)        count + = 1        print (' Worker running ') class MyThread (threading. Thread):    def start:        print (' start~~~~~~~~~~~~~~~ ')        super (). Start ()     def run (self):        Print (' run~~~~~~~~~~~~~~~~~ ')        super (). Run () t = MyThread (name= ' worker ', Target=worker)

　　

# T.start ()

T.run ()

The start () method calls the run () method, and the run () method can run the function.

These two methods appear to be functionally repetitive, but they cannot be left alone, as follows:

Import Threadingimport Time def worker ():    count = 0 while    True:        if (Count > 5):            Break        Time.sleep ( 1)        count + = 1        print ("Worker running")        print (Threading.current_thread (). Name) class MyThread (threading. Thread):    def start:        print (' start~~~~~~~~~~~~~ ')        super (). Start ()     def run (self):        Print (' run~~~~~~~~~~~~~~~ ')        super (). Run () t = MyThread (name= ' worker ', Target=worker) # T.start ()

　　

T.run () # Executes the start or Run method, respectively.

Start the thread with the Start method, start a new thread, named worker running, but use the Run method to start a thread, and not start a new thread, just call a normal function in the main thread.

Therefore, the startup thread uses the Start method to start multiple threads.

Multithreading

As the name implies, multiple threads, if there are multiple threads in a process, are multithreaded, implementing a concurrency.

Import Threadingimport Time def worker ():    count = 0 while    True:        if (Count > 5):            Break        Time.sleep ( 2)        count + = 1        print (' worker running ')        print (Threading.current_thread (). Name, Threading.current_thread ( ). Ident) class MyThread (threading. Thread):    def start:        print (' start~~~~~~~~~~~~~~ ')        super (). Start ()     def run (self):        Print (' run~~~~~~~~~~~~~~~~~~ ')        super (). Run ()  # See what the parent class is doing? T1 = MyThread (name= ' worker1 ', target=worker) t2 = MyThread (name= ' Worker2 ', Target=worker) T1.start () T2.start ()

You can see that worker1 and Worker2 are alternately executed.

Try the Run method:

Import Threadingimport timedef worker ():    count = 0 while    True:        if (Count > 5):            Break        Time.sleep ( 1)        count + = 1        print (' worker running ')        print (Threading.current_thread (). Name, Threading.current_thread ( ). Ident) class MyThread (threading. Thread):    def start:        print (' start~~~~~~ ')        super (). Start ()     def run (self):        print (' run~ ~~~~~~~~~~~ ')        super (). Run () T1 = MyThread (name= ' worker1 ', target=worker) t2 = MyThread (name= ' Worker2 ', target= Worker) # T1.start () # T2.start () T1.run () T2.run ()

Without opening a new thread, this is the normal function call, so execute T1.run () and then execute T2.run (), which is not multithreaded.

when using the start after the method starts the thread, there are multiple active threads in the process that work in parallel, that is, multithreading.

There is at least one thread in a process, and as the entrance to the program, this thread is the main path. A process has at least one main thread.

Other threads are called worker threads.

Thread Safety

Need to demonstrate in Ipython:

In [1]: Import threading   ...:  def worker (): ...: for   x in     Range (5):   ...:         print ("{} is running". Format (Threading.current_thread (). Name))   ...: ...:  for x in range (1, 5):   ...:     name = ' worker{} ' . Format (x) ...   :     t = Threading. Thread (Name=name, Target=worker) ...   :     t.start () ...   :

　　

You can see that in the run result, it was supposed to be a row of prints, but many strings were printed together, indicating that the print function was interrupted and interrupted by a thread switch.

The print function takes two steps, the first step is to print the string, the second line wraps, and the thread switches between them.

Description The print function is not a thread-safe function.

Thread Safety: threads execute a piece of code without producing an indeterminate result, and that code is thread-safe.

1. Do not let print wrap:

Import Threadingdef Worker (): for    x in range:        print (' {} is running\n '. Format (Threading.current_thread (). Name), end= ") for x in range (1, 5):    name = ' worker{} '. Format (x)    t = Threading. Thread (Name=name, Target=worker)    T.start ()

　　

A string is an immutable type, which can be an indivisible output as a whole. The end= ' function is to keep print output line-wrapping.

2, use logging.

The logging module in the standard library, the log processing module, thread safety, and the build Environment code all use logging.

Import Threadingimport Logging def worker (): for    x in range:        # print ("{} is running.\n". Format ( Threading.current_thread (). Name), end= ')        logging.warning (' {} is running '. Format (Threading.current_thread (). Name)) for x in range (1, 5):    name = ' work{} '. Format (x)    t = Threading. Thread (Name=name, Target=worker)    T.start ()

　　

Daemon Threads and Non-daemon threads

Note: Daemon here is not a daemon in Linux.

A process executes code on a thread, at least one main thread, and the other thread is a worker thread.

The main thread is the first one to start.

Parent thread: If thread A starts a thread b,a is the parent thread of B.

Child Thread: B is a sub-thread.

When constructing threads in Python, you can set the Daemon property, which must be set before the Start method.

In the __init__ method of the source code thread:

If daemon is not None:    self._daemonic = Daemon  # user set bool value. else:    self._daemonic = Current_thread (). daemonself._ident = None

　　

Thread Daemon Property, if the setting is the user's settings, otherwise take the current thread's daemon value.

The main thread is Non-daemon, or daemon=false.

Import timeimport Threading def foo ():    time.sleep (5) for    I in range:        print (i)

# The main thread is Non-daemon threads.

t = Threading. Thread (Target=foo, Daemon=false)

T.start ()

Print (' Main Thread Exiting ')

run the discovery thread T still executes, the main thread has finished executing, but has been waiting for the thread T.

Modify to T = Threading. Threading (Target=foo, daemon=true), Run the discovery main thread executes the program immediately ended , there is no waiting thread T.

Import Threadingimport logginglogging.basicconfig (Level=logging.info) #警告级别import time def Worker (): for    x in range (Ten):        time.sleep (1)        msg = ("{} is running". Format (Threading.current_thread ()))        Logging.info (msg)        t = Threading. Thread (target=worker1,name= "worker1-{}". Format (x), Daemon=false)        T.start ()        # t.join () def worker1 ():    For x in range:        time.sleep (0.3)        msg = ("￥￥￥￥￥{} is running". Format (Threading.current_thread ()))        Logging.info (msg) T = Threading. Thread (target=worker,name= ' worker-{} '. Format (0), daemon=true) T.start () # T.join () time.sleep (0.3) print (' ending ') Print (Threading.enumerate ())

　　

Conclusion:

Daemon=false runs the discovery that the child thread is still executing, the main thread has finished executing, but the main thread is waiting for the child thread to finish executing.

daemon=true running the discovery main thread finishes executing the program immediately.

Daemon Property: Indicates whether the thread is a daemon thread, this value must be set before start (), or a RuntimeError exception is thrown.

Isdaemon (): Whether it is a daemon thread.

Setdaemon: Set to daemon thread, must be set before the Start method.

Summarize:

The thread has a daemon property that can be explicitly set to TRUE or false, or not set, and the default value of None is set.

If you do not set daemon, take the current thread's daemon to set it. The double thread inherits the daemon value of the child thread, with the same effect as setting none.

The main thread is Non-daemon, or daemon=false.

All threads created from the main thread do not set the Daemon property, the default is Daemon=false, which is the Non-daemon thread.

The Python program exits when the Non-daemon thread is not alive, that is, only the daemon thread is left, and the main path can only be exited, otherwise the main thread can only wait.

The following program output:

Import Timeimport threadingdef bar ():    time.sleep (+)    print (' Bar ') def foo (): For    i in range:        print (i)    t = Threading. Thread (Target=bar, Daemon=false)    T.start () # The main thread is Non-daemon threads. T = Threading. Thread (Target=foo, Daemon=true) T.start () print (' Main threading Exiting ')

In the example above, there is no output bar for this string, how can the modification be printed out bar?

Import Timeimport Threading Def Bar ():    time.sleep (1)    print (' Bar ') def foo (): For    I in range (5):        Print (i )    T = Threading. Thread (Target=bar, Daemon=false)    T.start () # The main thread is Non-daemon threads. T = Threading. Thread (Target=foo, Daemon=true) T.start () time.sleep (1) Print (' Main threading Exiting ')

Look again at an example to see if the main thread is appropriate to end daemon threads.

Import timeimport Threading def foo (n): for    I in range (n):        print (i)        time.sleep (1) T1 = Threading. Thread (target=foo, args = (ten,), daemon=true)  # Swap 10 and 20 to see the effect. T1.start () t2 = Threading. Thread (target=foo, args = (), daemon=false) T2.start () time.sleep (2) print (' Main threading Exiting ')

The example above shows that if there are Non-daemon threads, the main thread exits, and does not kill all daemon threads until all Non-daemon threads have finished,

If there are daemon threads, the main thread needs to exit, ending all daemon threads and exiting.

Join method

Import timeimport Threading def foo (n): for    I in range (n):        print (i)        time.sleep (1) T1 = Threading. Thread (Target=foo, args= (), Daemon=false) T1.start () T1.join ()  # set Join.print (' Main Thread Exiting ')

After the Join method is used, the daemon thread executes, and the mainline friend exits.

Join (Timeout=none), is one of the standard methods of threading.

One thread calls the join method of another thread, and the caller is blocked until the called thread terminates.

A thread can be join multiple times.

The timeout parameter specifies how long the caller waits, without setting a timeout, waiting for the calling thread to end.

Whose join method is called, who is to join and who to wait for.

Python Concurrency and threading