Python multithreaded programming

Source: Internet
Author: User

1, multi-threaded development background

With the development of computers, both hardware and software are developing rapidly.

At the very beginning, the computer is only a CPU for instruction control and operation, the program executes is a process of a process, that is, the sequential execution of the way, all processes are queued, and then the CPU out of one of the processes, and then run.

In the hardware development, slowly developed into a few CPUs, and developed a few core CPUs, so that in the general server are four-core, and at least two, so that each time the server processing, is to carry out multi-process processing, it may be said that a CPU processing a process situation appears, Thus at this time, the program also developed a multi-process multithreading concept, in a process with multiple threads, and a program can run multiple threads.


In multi-process and multi-threaded main development of the goal, that is, to improve the performance of computing, so that programming here is called multithreaded programming, multi Threading.


2. Multithreading in Python

In Python, multithreading is present, but it is not the same as multithreading in other programming, that is, there are multiple threads in a process, but, unfortunately, only one thread is supported in Python, that is, multithreading in Python is still like a single-threaded operation.

There is a lock in Python, that is Gil,global interpreter lock, the global interpreter lock, which controls only one thread running at the same time. In Python, multithreading is performed in the following ways:


See the above situation, that is, in a process can have more than one thread, but at the same time there is only one thread to run, then why use multithreading it, with a thread is not good?


In Python, the best handling of multithreading is for I/O intensive operations, that is, in computationally intensive operations, it is not appropriate to use multithreading, because at the same time, only one thread is used to the CPU, but I/O is blocking the way, Thus Python multithreading is still used on I/O operations, such as the input of a large number of users such as IO operations.


If it is computationally intensive, then in Python it is more appropriate to use a multi-process approach, where there is no such restriction as a thread lock when using multiple processes in Python.


3. Differences between processes and threads

The English representation of the process is the processes, and the English language is thread.

The resources (mainly memory) consumed by the process are more than the resources consumed by the thread, the process has its own independent data space, and the thread is in a process, so all the threads are shared resources, the data communication between the threads is simple, and the inter-process communication needs to use the Ipc,inter Process Comminication, it may be necessary to use the socket for communication, communication between threads, can use a data structure of the container can be used to communicate, such as the data structure of the queue, this is a first-out data structure, can be used to communicate between threads.


When running the program, the operating system management process, there are threads in the process, in fact, the process management thread, if there are multiple threads, then the main thread will manage the child threads, when running the program, the first is a process a main thread, the main thread will manage the child threads.

In the operating system, we can directly query to the process number, which is the PID of the process, can directly kill a process, but we cannot directly kill a thread, the ID of the query process is as follows:

[[email protected] 521]# ps-ef|grep server |grep-v greproot     10981  1677  0 00:28 pts/0    00:00:00 python ser ver.py

In which the PID is 10981, so that the kill command can be used to directly kill the process, but can not find the PID of the thread, you can not kill a thread.


The end of the thread, either the thread runs its own end or the main thread kills the child thread.


One of the threads in the thread is the daemon, the daemon, which is running in the background, and the main thread does not wait for the daemon. The advantage of using daemons is that the main thread ignores the daemon thread, and if the main thread exits, the daemon will automatically destroy it, and the benefit of the daemon is that the main thread does not have to take the time to manage it.


4. Introduction to multi-threaded modules in Python

In Python, a multithreaded module is a thread module, there is a threading module, it is recommended to use the threading module, because threading belongs to the Advanced module, a lot of encapsulation, things are always more advanced better. In fact, the main reason is that in the thread module, you need to manage the synchronization between the sub-thread and the main thread of the problem, and in the threading module, has been related to the encapsulation, so that no tube synchronization problems.


5, multi-threaded examples

When using Python's multithreading, the basic steps are as follows:


The multithreading of the direct call function is as follows:

#!/usr/bin/env pythonfrom Threading Import Threadimport timedef Loop (name,seconds):    print ' Start loop ', Name, ' at: ', Time.ctime ()    time.sleep (1)    print ' End Loop ', name, ' at: ', Time.ctime ()    if __name__ = = ' __main__ ':    loops = [2,4]    nloops = range (len (loops))    threads = []    print ' starting at: ', Time.ctime () for    I in nloops:< C10/>t = Thread (target=loop,args= (I,loops[i],))        Threads.append (t) for    I in Nloops:        Threads[i].start ( )    for I in Nloops:        threads[i].join ()    print ' All-done at: ', Time.ctime ()

Here, the main thing to do in the loop function, thus creating multiple threads to execute the loop function, when the thread object is created, the direct use of the method called the function, that is, the first method of creating the thread, directly call the function we want to operate.

The second method uses a callable class object:

#!/usr/bin/env pythonfrom Threading Import Threadimport timedef Loop (name,seconds):    print ' Start loop ', Name, ' at: ', Time.ctime ()    time.sleep (1)    print ' End Loop ', name, ' at: ', Time.ctime ()    class ThreadFunc (object):    def __init__ (self,func,args,name= "):        self.name = name        Self.func = func        Self.args = args    def __call__ ( Self):        self.func (*self.args) if __name__ = = ' __main__ ':    loops = [2,4]    nloops = range (len (loops))    Threads = []    print ' starting at: ', Time.ctime () for    i in nloops:        t = Thread (Target=threadfunc (Loop, (I, Loops[i]), loop.__name__))        Threads.append (t) for    I in Nloops:        Threads[i].start () for    I in Nloops:        threads[i].join ()    print ' All do at: ', Time.ctime ()


When using callable class objects, primarily in the definition of a class, you need to define a special method __call__, and then when you create the thread, you need to instantiate the callable object using the target directly.


The third method is mainly sub-class, the code is as follows:

#!/usr/bin/env pythonfrom Threading Import Threadimport timedef Loop (name,seconds):    print ' Start loop ', Name, ' at: ', Time.ctime ()    time.sleep (1)    print ' End Loop ', name, ' at: ', Time.ctime ()    class ThreadFunc (Thread):    def __init__ (self,func,args,name= "):        Super (Threadfunc,self). __init__ ()        self.name = name        Self.func = Func        Self.args = args    def run (self):        self.func (*self.args) if __name__ = = ' __main__ ':    loops = [2,4]    Nloops = Range (len (loops))    threads = []    print ' starting at: ', Time.ctime () for    i in nloops:        t = Thread Func (Loop, (I,loops[i]), loop.__name__)        threads.append (t) for    I in Nloops:        threads[i].start ()    For i in Nloops:        threads[i].join ()    print ' All do at: ', Time.ctime ()

This method is much the same as the second method, mainly is to integrate the thread class when the subclass, and then modify the callable method to the Run method, in the thread class object, if you need to inherit, then the main is to rewrite the Run method.

The results of the implementation are as follows:

[[email protected] 522]# Python thread_demo.py starting at:sun may 03:44:49 2016start Loop 0  At:sun 22 03:44  : 2016start Loop 1  at:sun 03:44:49 2016end Loop 0 At:sun may be  03:44:50 2016end Loop 1  At:sun May 03:44:50 2016all done At:sun 22 03:44:50 2016

In the generally recommended approach, the third method, the subclass thread class, is used.

In this case, three loops are used, the main goal being to create a thread is to open all the threads in the second loop.


6. Thread Basic method

In the thread class of threading, the main method is the Run method, which is the associated method when using target initialization.

Thread instance, provide the target parameter, is the function that needs to execute, another args parameter is the function parameter, must be provided in the form of Ganso, if only one argument, must add a comma after one parameter, represent as Ganso.

Join indicates that the main thread does not need to wait for a child thread, and if not, you can not call the Join function, in the example above, call the Join function, so that the main thread waits for the child thread, after using the timeout parameter, it will indicate the maximum number of seconds for the main thread, and then proceed with the main thread. When the main thread ends, the child thread ends

Setdaemon is to set the thread as the daemon thread, the daemon thread represents a non-critical thread, the thread ends, and the daemon thread ends, you can use the method Isdaemon to determine if the daemon thread

GetName represents the name of the returned thread, SetName indicates the name of the set thread, IsAlive indicates whether the thread is still running

When you set up the daemon, you must set it before the Startup method is start.


7. Communication between threads

The specific process is as follows:


That is, the producer thread produces data written to the shared data, and then the consumer thread pulls the data out of the shared data.

The specific code is as follows:

#!/usr/bin/env pythonfrom Threading Import threadimport timefrom Queue import queuedef Writeq (queue,i): while True: Time.sleep (1) if Queue.full (): Time.sleep (1) else:queue.put (' xxx ') pri            NT ' producting object for Q ... ', idef READQ (queue,i): While True:time.sleep (1) if Queue.empty (): Time.sleep (1) print ' empty ' Else:val = Queue.get () print ' consumed object from Q.. Size now: ', queue.qsize (), ' Consumer ', IClass producter (Thread): def __init__ (self,func,args,name= "): Super (Pro ducter,self). __init__ () Self.name = name Self.args = Args Self.func = Func def run (self): s        Elf.res = Self.func (*self.args) #return self.resclass Consumer (Thread): def __init__ (self,func,args,name= "): Super (Consumer,self). __init__ () Self.name = name Self.args = Args Self.func = Func def run (SE LF): Self.res= Self.func (*self.args) #return self.resif __name__ = = ' __main__ ': q = Queue (+) for I in range: p = Producter (Writeq, (q, ' production%s '% i)) P.start () for I in Range (2): c = Consumer (READQ, (q, ' consume R%s '% i) C.start ()
In this, the main is to use the queue for inter-thread communication, in the above code, can be optimized, such as extracting public classes, derived sub-class producers and consumers.


8, the thread can not be created indefinitely

Test for the test thread can not be created indefinitely, the code is as follows:

[email protected] 522]# cat thread_test.py #!/usr/bin/env pythonfrom Threading Import Threaddef Testthread (): While    true:        passwhile true:    t = Thread (target=testthread)    T.start ()

is the unlimited creation of threads.

The Vmstat baseline is as follows:


The indicator is interpreted as follows:

R is a running queue, say more than 5, it's stressful.

In indicates the number of interrupts

CS indicates context switch

US represents user CPU time

Sy Indicates the CPU time of the system


After running for a period of time, it is found that the number of R of this procs is growing fast, so how many threads are allocated to the CPU

See, the CPU is very busy, the number of interrupts and context switching is very large, and the user's CPU time is more than.

At this time, log in to the system, you will find the system is very card, you can see the brain ... The CPU is simply not enough.


Return to normal after killing the process. Then remove the CPU limit, then you can use the following code:

#!/usr/bin/env pythonfrom Threading Import threadfrom time import Sleepi=0def testthread ():    print I while True:    i + = 1    t = Thread (target=testthread)    T.start ()

This test. There seems to be no thread limit.





Python multithreaded programming

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.