A simple example of using the map function to complete a Python parallel task

Python's parallel processing power is notoriously unsatisfactory. I think that if the standard parameters of threading and Gil are not taken into account (they are mostly legal), the reason is not because the technology is not in place, but because of the inappropriate use of our methods. Most of the textbooks on Python threading and multi-process are excellent, but the content is tedious and lengthy. They do have a lot of useful information at the beginning, but they don't always involve parts that really improve their daily work.

Classic examples

Popular search results on DDG with the keyword "python threading tutorial (Python Threading Tutorial)" indicate that the examples given in almost every article are the same class + queue.

In fact, these are the following code examples that use Producer/consumer to handle thread/multi-process:

#Example. py "Standard Producer/consumer threading Pattern" "Import timeimport Threadingimport Queue class Consumer ( Threading. Thread): Def __init__ (self, queue): Threading. Thread.__init__ (self) self._queue = Queue def run (self): when True: # queue.get () blocks the current thread      Until # an item is retrieved. msg = Self._queue.get () # Checks If the current message is # the "Poison Pill" if Isinstance (msg, str) and       msg = = ' quit ': # If so, exists the loop broke # "Processes" (or in our case, prints) the queue item    print "I ' m a thread, and I received%s!!"% msg # always be friendly!  print ' Bye byes! ' def Producer (): # Queue is used to share items between # the threads. Queue = Queue.queue () # Create An instance of the worker worker = Consumer (Queue) # Start calls the internal run () met Hod to # Kick off the Thread Worker.start () # variable to keep track of when we started start_time = Time.time () # W HilE under 5 seconds. While Time.time ()-Start_time < 5: # "Produce" a piece of work and stick it in # The queue for the Consumer to P Rocess queue.put (' Something at%s '% Time.time ()) # Sleep a bit just to avoid an absurd number of messages TIME.S  Leep (1) # This "poison pill" method of killing a thread. Queue.put (' Quit ') # Wait for the thread to close worker.join () if __name__ = = ' __main__ ': Producer ()

Hmm .... It feels a bit like java.

I don't now want to explain that using producer/consume to solve a thread/multi-process approach is wrong-because it's definitely correct, and in many cases it's the best approach. But I don't think it's the best choice for writing code.

Where it's the problem (personal view)

First, you need to create a boilerplate type of bedding class. You then create a queue that accomplishes the task by passing objects and supervising both ends of the queue. (If you want to implement the exchange or storage of data, it usually involves the participation of another queue).

The more workers, the more problems.

Next, you should create a pool of worker classes to increase the speed of Python. The following is a good approach given by IBM Tutorial. This is also a common way for the program staff to retrieve web pages using multithreading.

#Example2. py "A more realistic thread pool example" "Import Timeimport threadingimport Queueimport Urllib2 class consume R (Threading. Thread): Def __init__ (self, queue): Threading. Thread.__init__ (self) self._queue = Queue def run (self): while true:content = Self._queue.get () if Isin  Stance (content, str) and content = = ' quit ': Break response = urllib2.urlopen (content) print ' Bye byes! ' def Producer (): urls = [' http://www.python.org ', ' http://www.yahoo.com ' http://www.scala.org ', ' http://www.google.co  M ' # etc.. ] Queue = Queue.queue () worker_threads = Build_worker_pool (queue, 4) start_time = Time.time () # ADD the URLs to Proce SS for URLs in Urls:queue.put (URLs) # ADD The poison pillv for worker in Worker_threads:queue.put (' Quit ') for W Orker in Worker_threads:worker.join () print ' done!    Time taken: {} '. Format (Time.time ()-start_time) def build_worker_pool (queue, size): Workers = [] for _ in range (size): Worker =Consumer (queue) Worker.start () workers.append (worker) return workers if __name__ = ' __main__ ': Producer () 

It does work, but how complex the code is! It includes the initialization method, the thread-tracking list, and the nightmare of people who are as prone to error as I do on deadlock issues-a large number of join statements. And these are just a tedious start!

What have we done so far? There's basically nothing. The above code is almost always just passing. This is a very basic method, very easy to make mistakes (damn, I just forgot to call the Task_done () method on the queue object (but I didn't bother to change it)), the price is very low. Fortunately, we still have a better way.

Description: Map

Map is a great little feature, and it is also the key to the fast running of Python parallel code. Explain to unfamiliar people that map is from the functional language Lisp. The map function can map another function in order. For example

URLs = [' http://www.yahoo.com ', ' http://www.reddit.com ']results = Map (urllib2.urlopen, URLs)

Here, the Urlopen method is called to return all the call results sequentially and store them in a list. Just like:

results = []for URL in URLs:  results.append (Urllib2.urlopen (URL))

Map processes these iterations sequentially. Call this function and it will return to us a simple list of the results stored sequentially.

Why is it so powerful? As long as there is a suitable library, map can make parallel running very smooth!

There are two libraries that can support the parallel library through the map function: One is multiprocessing, the other is a little-known but powerful sub-file: Multiprocessing.dummy.

Off topic: What is this? Have you ever heard of dummy multi-process library? I have only recently known. It is only mentioned in the multi-process documentation. And that sentence is probably to let you know there is such a thing. I dare say that the consequences of such a near-selling approach are disastrous!

Dummy is the clone file of the multi-process module. The only difference is that the multi-process module uses the process, while dummy uses the thread (which, of course, has all the common limitations of Python). That is, the data is passed from one to the other. This makes the data easy to forward and bounce between the two, especially for exploratory programs, because you don't have to determine whether the framework calls are IO or CPU mode.

Ready to start

To accomplish parallelism through the map function, you should first import the modules that contain them:

From multiprocessing import poolfrom multiprocessing.dummy import Pool as ThreadPool

Re-initialize:

Pool = ThreadPool ()

This simple sentence can replace all the work of our Build_worker_pool function in example2.py. In other words, it creates a number of effective workers, launches them to prepare for the next work, and stores them in different locations for ease of use.

The pool object requires some parameters, but most importantly: the process. It determines the number of workers in the pool. If you don't fill it out, it will default to your computer's kernel value.

If you use a multi-process pool in CPU mode, the larger the number of cores, the faster (and many other factors). However, when working on a thread or dealing with network bindings, the situation can be complicated so the exact size of the pool should be used.

Pool = ThreadPool (4) # Sets the pool size to 4

If you run too many threads, switching between threads can be a waste of time, so you'd better be patient with the most appropriate number of tasks.

Now that we've created the pool object, we're going to have a simple parallel program, so let's re-write the URL opener in example2.py!

Import urllib2from multiprocessing.dummy import Pool as ThreadPool urls = [  ' http://www.python.org ',  '/HTTP/ www.python.org/about/',  ' http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html ',  ' HTTP// www.python.org/doc/',  ' http://www.python.org/download/',  ' http://www.python.org/getit/',  ' HTTP// www.python.org/community/',  ' https://wiki.python.org/moin/',  ' http://planet.python.org/',  ' https:/ /wiki.python.org/moin/localusergroups ',  ' http://www.python.org/psf/',  ' http://docs.python.org/ devguide/',  ' http://www.python.org/community/awards/'  # etc..  ] # Make the Pool of Workerspool = ThreadPool (4) # Open the URLs in their own threads# and return the Resultsresults = Pool.map (urllib2.urlopen, URLs) #close t He pool and wait for the work to Finishpool.close () Pool.join ()

Look at it! This time the code only took 4 lines to complete all the work. 3 of these sentences are still simple fixed notation. Call map to complete the 40 lines of our previous example! In order to show the difference between the two methods more vividly, I also give them time to run separately.

# results = []# for URL in urls:#  result = Urllib2.urlopen (URL) #  results.append (Result) # #-------VERSUS------- # # #-------4 pool-------# # pool = ThreadPool (4) # results = Pool.map (Urllib2.urlopen, URLs) # #-------8 Pool------ -# # Pool = ThreadPool (8) # results = Pool.map (Urllib2.urlopen, URLs) # #-------Pool-------# # pool = ThreadPool (13 ) # results = Pool.map (Urllib2.urlopen, URLs)

Results:

# single            thread:14.4 seconds#               4 pool:  3.1 seconds#               8 pool:  1.4 seconds#               Pool:  1.3 Seconds

Pretty good! It also shows why you should carefully debug the size of the pool. Here, as long as it is greater than 9, it can make it run faster.

Example 2:

Generate thousands of thumbnails

Let's do it in CPU mode! I often have to deal with a lot of image folders in my work. One of its tasks is to create thumbnails. This is already a very mature method in parallel tasks.

Basic single-threaded creation

Import Osimport PIL from multiprocessing import poolfrom PIL import Image SIZE = (75,75) save_directory = ' thumbs ' def get_ Image_paths (folder):  return (Os.path.join (folder, F) for      F in Os.listdir (folder)      if ' jpeg ' in f) def Create_ Thumbnail (filename):  im = Image.open (filename)  im.thumbnail (SIZE, Image.antialias)  base, fname = Os.path.split (filename)  Save_path = Os.path.join (base, Save_directory, fname)  Im.save (save_path) if __name__ = = ' __main__ ':  folder = Os.path.abspath (    ' 11_18_2013_r000_iqm_big_sur_mon__e10d1958e7b766c3e840 ')  Os.mkdir (Os.path.join (folder, save_directory))   images = get_image_paths (folder) for   image in images:       Create_thumbnail (Image)

This is a bit difficult for an example, but essentially, it's a way to pass a folder to the program and then grab all of the pictures and eventually create and store thumbnails in their respective directories.

My computer took 27.9 seconds to process about 6000 images.

If we were to call map in parallel instead of a for loop:

Import Osimport PIL from multiprocessing import poolfrom PIL import Image SIZE = (75,75) save_directory = ' thumbs ' def get_ Image_paths (folder):  return (Os.path.join (folder, F) for      F in Os.listdir (folder)      if ' jpeg ' in f) def Create_ Thumbnail (filename):  im = Image.open (filename)  im.thumbnail (SIZE, Image.antialias)  base, fname = Os.path.split (filename)  Save_path = Os.path.join (base, Save_directory, fname)  Im.save (save_path) if __name__ = = ' __main__ ':  folder = Os.path.abspath (    ' 11_18_2013_r000_iqm_big_sur_mon__e10d1958e7b766c3e840 ')  Os.mkdir (Os.path.join (folder, save_directory))   images = get_image_paths (folder)   pool = Pool ()    Pool.map (create_thumbnail,images)    pool.close ()    pool.join ()

5.6 Seconds!

This greatly improves the speed of running a few lines of code. This method can also be faster, as long as you run the CPU and IO tasks separately with their processes and threads-but also often cause deadlocks. In a word, considering the practical function of map and the lack of human thread management, I think it is a beautiful, reliable and easy way to debug.

Well, the article is over. Line to complete the parallel task.

