Comparison of Python multi-thread image capturing efficiency,

Comparison of Python multi-thread image capturing efficiency,

Purpose:

It is to learn how python multithreading works, and to view the Efficiency Comparison of multithreading by capturing 400 images, such as IO-intensive applications.

Import requestsimport urlparseimport osimport timeimport threadingimport Queuepath = '/home/lidongwei/scrapy/done' # path ='/home/lidongwei/scrapy/cc.txt 'fetch _ interval = '/home/lidongwei /scrapy/owan_imgs/'# Read and save the file, and then read the 400 urlswith open (path) in the file) as f: urls = f. readlines () urls = urls [: 400] # Use Queue for thread communication, because the Queue is thread-safe (that is, the Queue has a lock by default) q = Queue. queue () for url in urls: q. put (url) start = time. time () def fetch_img_func (q): while True: try: # non-blocking read queue Data url = q. get_nowait () I = q. qsize () failed t Exception, e: print e break; print 'current Thread Name Runing % s... 11' % threading. currentThread (). name url = url. strip () img_path = urlparse. urlparse (url ). path ext = OS. path. splitext (img_path) [1] print 'handle % s pic... pic url % s' % (I, url) res = requests. get (url, stream = True) if res. status_code = 200: save_img_path = '% s % s' % (fetch_img_save_path, I, ext) # Save the downloaded image with open (save_img_path, 'wb') as fs: for chunk in res. iter_content (1024): fs. write (chunk) print 'Save % s pic '% I # You can open multiple threads to test different effects t1 = threading. thread (target = fetch_img_func, args = (q,), name = "child_thread_1") # t2 = threading. thread (target = fetch_img_func, args = (q,), name = "child_thread_2") # t3 = threading. thread (target = fetch_img_func, args = (q,), name = "child_thread_3") # t4 = threading. thread (target = fetch_img_func, args = (q,), name = "child_thread_4") t1.start () # t2.start () # t3.start () # t4.start () t1.join () # t2.join () # t3.join () # t4.join () end = time. time () print 'done % s' % (end-start)

Lab results

400 Images

4 threads Done 12.4431338313 threads Done 12.9201757908 2 threads Done 32.86282992361 threads Done 54.6115460396

Summary

Python comes with a GIL lock and does not actually implement multi-thread parallel execution. The GIL lock will be released when the thread is blocked, and the waiting thread can activate the work, and so on, greatly improving the efficiency of IO blocking applications.

Articles you may be interested in:

• Python multi-thread programming (6): reentrant lock RLock
• Python multi-thread programming (5): deadlock Formation
• Python multi-thread programming (4): Use Lock mutex Lock
• Analysis on multithreading and program lock in Python
• Python multi-thread threading. Lock usage example
• Python thread learning example
• Python implements simple multi-thread task queue
• Python multi-thread, asynchronous, and multi-process crawler implementation code
• Using Queue and Condition for Thread Synchronization in Python
• A Brief Introduction to the creation of threads and the use of locks in Python Programming