Python Multithreading anatomy

First look at a chestnut:

Let's take a look at the I/O cheats thread, for a chestnut-crawler, below is the climb down the picture with 4 threads to write a file

1 #!/usr/bin/env python2 #-*-coding:utf-8-*-3  4 ImportRe5 ImportUrllib6 ImportThreading7 ImportQueue8 ImportTimeit9 Ten defgethtml (URL): OneHtml_page =urllib.urlopen (URL). Read () A     returnHtml_page -   - #extract the URL of a picture in a Web page the defgetUrl (HTML): -Pattern = R'src= "(http://img.*?)"'  #Regular Expressions -Imgre =re.compile (pattern) -Imglist = Re.findall (imgre, HTML)#Re.findall (pattern,string) finds all successful strings in string, returns the value as a list +     returnimglist -   + classgetimg (Threading. Thread): A     def __init__(Self, queue, thread_name=0):#thread common one queue atThreading. Thread.__init__(self) -Self.queue =Queue -Self.thread_name =Thread_name -Self.start ()#Start Thread -   -     #using queues for interprocess communication in     defRun (self): -         GlobalCount to          while(True): +Imgurl = Self.queue.get ()#call the Get () method of the queue object to remove from the team header and return an item -Urllib.urlretrieve (Imgurl,'E:\mnt\girls\%s.jpg'%count) theCount + = 1 *             ifself.queue.empty (): $                  BreakPanax NotoginsengSelf.queue.task_done ()#When a consumer thread calls Task_done () to indicate that the project has been retrieved and all the work has been done, the total number of unfinished tasks is reduced.  -Imglist = [] the defMain (): +     Globalimglist AURL ="http://huaban.com/favorite/beauty/"  #the page address to crawl theHTML =gethtml (URL) +Imglist =getUrl (HTML) -   $ defmain_1 (): $     GlobalCount -Threads = [] -Count =0 theQueue =Queue.queue () -     #join All tasks to the queueWuyi      forImginchimglist: the queue.put (IMG) -     #Multi-threaded crawling to pictures Wu      forIinchRange (4): -Thread =getimg (queue, i) About threads.append (thread) $     #block threads until thread execution is complete -      forThreadinchThreads: - Thread.Join () -   A if __name__=='__main__': + Main () thet =Timeit. Timer (main_1) -     PrintT.timeit (1)

View Code

4 Thread Execution time: 0.421320716723 seconds

Modify the main_1 switch to single-threaded:

1 defmain_1 ():2     GlobalCount3Threads = []4Count =05Queue =Queue.queue ()6     #join All tasks to the queue7      forImginchimglist:8 queue.put (IMG)9     #Multi-threaded crawling to picturesTen      forIinchRange (1): OneThread =getimg (queue, i) A threads.append (thread) -     #block threads until thread execution is complete -      forThreadinchThreads: theThread.Join ()

View Code

Single-Threaded execution time: 1.35626623274 seconds

Let's look at one more:

1 #!/usr/bin/env python2 #-*-coding:utf-8-*-3 ImportThreading4 ImportTimeit5 6 defcountdown (N):7      whilen >0:8N-= 19 Ten defTask1 (): OneCOUNT = 100000000 AThread1 = Threading. Thread (Target=countdown, args=(COUNT,)) - Thread1.start () - Thread1.join () the  - defTask2 (): -COUNT = 100000000 -Thread1 = Threading. Thread (Target=countdown, args= (COUNT//2,)) +Thread2 = Threading. Thread (Target=countdown, args= (COUNT//2,)) - Thread1.start () + Thread2.start () A Thread1.join () at Thread2.join () -  - if __name__=='__main__': -T1 =Timeit. Timer (TASK1) -     Print "countdown in one thread", T1.timeit (1) -t2 =Timeit. Timer (TASK2) in     Print "countdown in both thread", T2.timeit (1)

View Code

Task1 is a single threaded, Task2 is a double-threaded execution result on my 4-core machine:

Countdown in one thread 3.59939150155

Countdown in both thread 9.87704289712

God, dual threads are twice times slower than single-threaded computations, which is why, because countdown is CPU intensive (COMPUTE)

I/O intensive tasks: When the thread does I/O processing, the Gil is freed, and the other threads get the Gil, and when the thread does the I/O operation, it releases the Gil, and so forth;

CPU-intensive tasks: multi-core multithreading is worse than single-core multithreading, because of single-core multi-threading, each time the Gil is freed, which thread wakes up can get to the Gil lock, so it can be executed seamlessly (the essence of single core multithreading is sequential execution), but multicore, CPU0 release Gil, Other threads on the CPU will compete, but the Gil may be immediately CPU0 (possibly more than one thread on the CPU0), causing several other CPUs to wake up to wait until the switch time and then go to the scheduled state, causing thread bumps (thrashing), result in lower efficiency.

Python Multithreading anatomy