Parallel thinking [I]

ArticleDirectory

• Statement:
• Data Parallelism
• Parallel Tasks
• Pipelines/pipelines (merging data and parallel tasks)
• Hybrid Solution
• Parallel Implementation

Statement:

This article mainly describes how to make it secure and scalable.ProgramParallelism. Today, with multiple cores, we may need to think more about how to write a good parallel program. A considerable amount of content in this article comes from Intel threading building blocks. Although it is intended for C ++, it is generally used in general principles and will be used when it is associated with languages and platforms.. net.

Every software developer has to deal with parallel programming. Before and now, when we finish the task, we will first consider selecting the bestAlgorithmLanguage. But now we must first consider the inherent concurrency of the task. In turn, this will affect our choice of algorithms and implementations. If you try to consider parallelism at the end, it is better not to think about parallelism. The program cannot work well either.

In reality, we are thinking in parallel every day. Here are two examples:

1. Long wait. For example, if you are ill, go to the hospital to see a doctor. When the number is reached before the outpatient service, it takes a long time to see the doctor. At this moment, I will not be silly to wait. In the absence of serious problems, I will wait for a visit while doing something shorter. For example, openMP3Listen to music, chat with patients, and so on.
2. A lot of repetitive work. When there is a heavy repetitive task at hand, and there are many friends who are familiar with it, I certainly hope these people will help you to complete the task faster. For example, to migrate data, the network administrator must install the same software on each machine.

We are no stranger to parallelism. In fact, parallelism is a natural way of thinking. But it seems that developers do not often use this method. Once we focus on and use parallel programming, we will think about parallelism. At that time, we will first consider the concurrency of the entire project and then consider how to code it.

How should we implement Program parallelism? This article describes an important way of thinking: Parallel decomposition.

Applications have multiple parallel methods, which are described below.

Data Parallelism

1. It is a typical example of data parallelism: it has a large amount of data, and the data adopts the same operation to convert each piece of data.

Figure 1

Figure 1 indicates that each lowercase English letter in the dataset is converted into a corresponding uppercase letter. This simple example shows the dataset to be operated on and the conversion operation applied to each element at the same time. Developers who code for the supercomputer most like this problem, because it is really easy to make it parallel.

Parallel Tasks

Data parallelism is often limited by the amount of data to be processed. In reality, the number of CPU cores cannot cope with huge data growth, even if the CPU can execute multiple commands in parallel in disorder. In addition, many times the bottleneck is not at the CPU, and the memory and cache also have an important impact. We can even involve GPU in processing. At this time, we will turn to parallel tasks (2 ).

Parallel Tasks mean a large number of different, independent tasks associated with shared data. Figure 2 shows that some mathematical calculation rules are applied to the same dataset to obtain its values, which are relatively independent. In other words, we can calculate the value independently, without the impact of other computations.

Figure 2

Pipelines/pipelines (merging data and parallel tasks)

It is much harder to find that pure task parallelism is more difficult than pure data parallelism. Most of the time, when you find that a task is running at the same time, it is also a specific pipeline/pipeline. Many independent tasks need to be applied to data streams. Each item is processed in stages, as shown in section 3 from A to.

Figure 3

If you use pipelines or pipelines, data flow A can be processed more quickly. Because different items can be processed in parallel at different stages. Pipelines/pipelines are more advanced than other processing methods: they can change data or skip certain steps for selected items. The assembly line of an automobile factory is a good example of a pipeline/assembly line. 4:

Figure 4

Hybrid Solution

Consider how to send a letter after writing your deep thoughts on the letter. It takes about six steps: Fold the letter paper, load the letter paper into an envelope, seal the envelope, write the address on the front of the envelope, attach a stamp, and deliver the mail. If you have six people to complete a large number of envelope loading and delivery tasks, what would you do? The natural response is to schedule one person to complete each phase following the six steps above. 5:

Figure 5

With reference to the data parallelism mentioned above, you may want to separate these six individuals and complete all the steps by themselves. 6:

Figure 6

If each person works in different locations and is separated from each other, figure 6 is clearly the right choice. This is the so-called "coarse-grained parallelism", because there is little interaction between each other's tasks (they only get together when they take the envelope, and then turn around to do the work, and even include delivery ). In addition, 5 shows "fine-grained parallelism" because they often interact with each other (each envelope is transmitted by the last person and then transmitted to the next person ).

The program may not be completely realistic, but sometimes it is useful to be close enough. In this example, the fourth step (write the address on the front of the envelope) may keep three people busy, but the first two steps and the last two steps only need one person to catch up. Figure 7 shows the workload of each step, while Figure 8 shows a real hybrid solution of Data Parallel and task parallel.

Figure 7

Figure 8

Parallel Implementation

The above example can be expressed in the following two steps:

1. Allocate tasks (around the workload ).
2. At the beginning, each person made part of six tasks, but was happy to split the established tasks so that two or more people could work together.

Then how should we program in parallel? First, consider where your program can be parallel. Secondly, perform parallel decomposition based on the actual situation. If we are lucky, we may only encounter data parallelism, which is very simple. For example, parallel iteration. When a complex scenario is encountered, the comparison based on the above classification is the most troublesome and only a hybrid solution. Use hardware parallelism as much as possible, which greatly eliminates the need to use the lock mechanism for synchronization and is more efficient.

To be continued...