Python parallelism (parallel)

Source: Internet
Author: User

Python parallelism (parallel)
Introduction

You can check it out first.Difference between Concurrent and Parallel

Before talking about parallelism, words such as multithreading, multi-process, thread/process synchronization, and thread/Process Communication will always pop up.

So why do we need to synchronize and communicate with each other?

Communication: multi-thread/process communication. For example, when I call you, what I say, what you answer, or when I say, you only listen. It focuses on data transmission.

Synchronization is actually relative to the shared memory. For example, if we modify the data of the shared object in the same place at the same time, this will lead to data tampering and will not get the desired result, in this case, synchronization is required. It is based on sharing the same object.

 

In the software world, there are many patterns and frameworks, such as design patterns and MVC frameworks. Do parallel programming also have some patterns?

Four Common parallel Modes

Now let's take a look at the thread/process synchronization, whether they are based on the shared memory mode; thread/Process Communication corresponds to the message transmission mode.

Note: Data parallelism separates data into multiple groups for independent data processing.

Four phases of designing concurrent programs (PCAM Design Methodology ):

Case Study

This case is implemented using Python multi-process. The example of inter-process communication on the official website is Exchanging objects between processes. The example of process Synchronization on the official website is Synchronization between processes and Sharing state between processes.

 

Design concurrent programs.

First, the task decomposition: one process stores data, and multiple processes fetch data and perform computation.

Second, Process Communication: takes the queue for access process communication.

Finally, combination: how can we combine the data calculated by multiple processes? The Value of the shared memory data of multiple processes is used. If multiple processes operate on the shared data, synchronization is required.

1 import multiprocessing 2 3 def make_data (queue, num, work_nums): 4 for I in range (num): 5 queue. put (I) 6 for I in range (work_nums): 7 queue. put (None) 8 9 def handle_data (queue, pai_value, lock): 10 while True: 11 data = queue. get () 12 if data is None: 13 break14 lock. acquire () 15 pai_value.value = pai_value.value + data16 lock. release () 17 18 if _ name _ = "_ main _": 19 queue = multiprocessing. queue () #20 used for inter-process communication pai_value = multiprocessing. value ("I", 0) #21 lock = multiprocessing used for inter-process sharing. lock () # When memory is shared between processes, adopt lock synchronization mechanism 22 num = 10000 #23 work_nums = 5 # Number of work processes 24 sub_process = [] # process data process set 25 26 master_process = multiprocessing. process (target = make_data, args = (queue, num, work_nums,) # generate the data Process 27 for I in range (work_nums): 28 sub_process1 = multiprocessing. process (target = handle_data, args = (queue, pai_value, lock,) 29 sub_process.append (sub_process1) 30 31 master_process.start () 32 for p in sub_process: 33 p. start () 34 35 master_process.join () 36 for p in sub_process: 37 p. join () 38 39 # result comparison 40 result = 041 for I in range (num): 42 result = result + i43 print ("result shocould be" + str (result )) 44 print ("fact is" + str (pai_value.value ))

 

Output result:

Result shocould be 49995000

Faces is 49995000

 

 

 

 

 

 

Refer:

1. Python Parallel Programming Cookbook

2. General design process of Parallel Algorithms

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.