"Programming in Go" The seventh chapter concurrent programming translation

Chinese translation	Go programming
Original name of foreign language	Programming in Go
Source of foreign language version		The States on recycled paper at RR Donnelley in Crawfordsville, Indiana.
President Seventh Chapter concurrent Programming 7.1 Main Concepts 7.2 Examples 7.2.1: Filter 7.2.2: Concurrency Lookup 7.2.3: Thread-Safe tables 7.2.4:apache Report 7.2.5: Looking for duplicates Concurrent programming enables developers to implement parallel algorithms, as well as to take advantage of multiprocessor and multi-core write programs. But in most mainstream languages, such as c,c++ and Java, relatively single-threaded programs, it is difficult to implement, maintain, and debug concurrent applications. Further, it is not always possible to separate processing, it is worthwhile to use multithreading. In any case, it is not always possible to achieve the expected performance advantage because of the overhead of the thread itself or simply because it is easier to make mistakes in multithreaded programs. One way is to completely avoid threads. For example, we can transfer the burden to the operating system through multiple processors. However, the disadvantage is that the operating system gives us the responsibility to control inter-process communication, most of which is more expensive than shared memory concurrency. There are three parts to the go solution. First, go provides a high degree of support for concurrent programming, making it easier to do it correctly, and second, the process of concurrency is done in a more lightweight way than threads, and thirdly, the automatic garbage collection mechanism sometimes eases the complexity of programmers ' memory management required by concurrent programs. Go is based on the CSP (Message Queuing process), and the built-in advanced application interface is provided to the Concurrency program implementation. This means that the display is locked, which avoids all the attention needed to lock and unlock at the right time, and uses the sending and receiving data to synchronize through a thread-safe channel. This greatly simplifies the writing of concurrent programs. On the other hand, dozens of threads allow a typical desktop computer to load, and the same machine can handle hundreds or even tens of thousands of processes with ease. The go approach allows programmers to think about what they want concurrent programs to achieve, rather than locking and other underlying details. Most other languages support very low-level concurrency (atomic increase, contrast, and exchange), and some low-level tool mutexes, and no other mainstream language like go provides such built-in high-level concurrency support. (perhaps except for additional libraries, not part of the language)
In addition to the high degree of support concurrency is the topic of this chapter, go also offers the same low-level functionality as other languages. At the lowest level, the Sync/atomic package of the standard library provides functions to complete atomic addition, comparison, and exchange operations. These advanced functions are designed to support thread-safe synchronization algorithms and data structure implementations that they do not apply to application programmers. The Go Sync Pack provides common low-level concurrency primitives: wait conditions and mutexes. These are similar to his peers in other languages, so application programmers are often forced to use them. Go programmer concurrent Programming will use the GO advanced Tools-Channel and co-process. In addition, no matter how many times the call, sync. The Once type can only be used to call a function, sync. The Waitgroup type provides an advanced synchronization mechanism, which you'll see later. We have covered the basic syntax as well as the use of channels and co-processes in the fifth chapter. These are not repeated here and are assumed to have been understood, so rereading, at least skimming, is useful for follow-up. This chapter begins by describing some of the main concepts that go concurrency becomes inside. Then this chapter shows five complete programs explaining go concurrency programming, showing some basic usage. The first example shows how to create pipelines, with each part of the pipeline performing the maximum throughput of their respective processes. The second example shows how to separate the work into a certain number of processes and output the results of their independence from one another. The third example shows how to create a thread-safe data structure without using visible locks or low-level basic elements. The fourth example shows how to accomplish each part of the work independently in a certain number of processes and merge the results together in three different ways. The fifth example shows how to rely on the process to create a certain number of threads, and how to combine the work of these processes into a single result set. 7.1 Main Concepts In concurrent programming, we typically want to separate processes that require more than one or more threads (as opposed to the main coprocessor), and either when the output is computed or the result output is merged at the end. Even with the advanced concurrency method of go, there are pitfalls that we must avoid. One of the pitfalls is that the program will end immediately without producing any results. At the moment the main process terminates, the Go program terminates automatically, even if other processes are still running, so we must be careful enough to allow the main process to keep enough time to complete all the work. Another pitfall we have to avoid is deadlock. One form of this problem is essentially the opposite of the first trap: the main thread and all processing threads remain active, even if all the work is done. This is usually a failure to finish processing the report. Another deadlock is two different threads that use locks to protect resources and request the same lock at the same time, as shown in Figure 7.1. This deadlock only occurs when a lock is used, so this is a common risk for other languages, but rather less than go now, because the Go app can use the channel to avoid the use of locks.
The usual way to avoid premature termination and not to end is to let the boiling process wait for a "complete" channel to report that the work has been done (we'll see it at once, or we can see it in 7.2.2,7.2.4). (You can also send a flag value in the final "result", but this is awkward relative to the other method) Another way to avoid traps is to use sync. Waitgroup waits for all the processing to report their end. However, use sync. The waitgroup itself can also lead to deadlocks, especially when all processing threads are blocked (for example, waiting to be received from the channel) and sync is present in the main process. The invocation of waitgroup.wait (). We'll see how to use sync later. Waitgroup. (7.2.5) In go, even if we just use a channel without a lock, we can still create a deadlock. For example, suppose we have a series of processes that can access each other, execute functions (such as sending requests to each other). Now if one of the requested functions is sent to one of the executing threads, such as sending some data, we will generate a deadlock. Shown in Figure 7.2 (we will see that this deadlock is possible in 337,340) The channel provides a means of communication that is running in parallel, without locking. (at the bottom of the lock is also available, but these we do not care about their own implementation details.) When a channel communication occurs, the sending and receiving channels (and their respective co-processes) are synchronized at the same time. The default channel is bidirectional, that is, we can send data into the channel and get the data through them. However, it is quite common to put the channel into a structure or pass the channel to the parameter as a single channel, that is, it can only send data or receive data. In these cases, we are able to differentiate the direction of the channel by the semantics of the expression (and by forcing the compiler to check). For example, type Chan<-type is a channel that sends only messages, and type <-chan is a channel that receives only messages. In the previous chapters we do not use these syntaxes, because we do not need, we can always use the Chan type to replace, and there are many other to learn. But from now on, we will use one-way channels when appropriate, because they generate additional compile time checks, and best practices. Sending values like Bool,int,float64 through the channel is intrinsically safe because these are copies, so there is no risk of inadvertent concurrency for the same values. Similarly, sending string types is also safe because they are immutable. Sending a pointer or reference (such as slice or map) through the channel is not inherently secure, because the value pointed to or referenced can be changed at the same time as the process in which it was sent, or the process at which it was received, without the desired result.
So, when a pointer or reference is coming, we have to make sure that they can only be used by a single process at the same time, which means that the right to use must be serialized. The exception is that in the file it is particularly stated that it is safe to pass pointers, such as the same *regexp. Regexp can be used safely in multiple processes if we need to change the state of a value because there is no method to use the value. One of the methods for serializing access is to use a mutex. Another approach is to apply a guideline where pointers or references are sent only once, and once sent, the sender will no longer use it. This allows the receiver to receive the value pointed to or referenced at any time, and provides the same guideline for sending pointers or references to the sender. (We will see an example based on this approach; 7.2.4.3. The downside is that the approach based on this approach is needed for training. The third method of using pointers or referencing work security is to provide an exit method that cannot change the value that is pointed to or referenced to, rather than the exit method being able to perform the change. Such pointers or references can be transmitted and accessed at the same time through their export methods, and only one process uses their non-egress methods (for example, in their own case, the package is explained in the Nineth chapter). It is also possible to send an interface value, that is, a value that satisfies a particular interface, through a channel. The value of the read-only interface can be safely used in any number of threads (unless the file description is not possible), but the interface with the value, which contains the state that can change the value, must be treated like a pointer, accessing serialization. For example, when we use image. The Newrgba () function creates a new picture and will get a *image. RGBA. This type satisfies the image at the same time. The Image interface (only the Get method, which is read-only), and draw. Image interface (has an image.) The Image method plus a set () method). So, transfer the same *image. The RGBA value is safe in multiple threads and provides us with a transmission to receive an image. The function of the image. (Unfortunately, this security will be compromised if the receiving method uses an assertion, draw.) Image interface, so it is very sensible not to allow such things. When we want to use the same *image that can be changed in multiple processes. Rgba value, we should either send *image. Rgba or draw. Image, either one of us is going to make sure that it is serialized. One of the simplest ways to use concurrency is to use a co-process to prepare for these tasks, and then another to do the work, allowing the main process and some channels to schedule everything. For example, here's how we create a "jobs" channel in the main process and a "done" channel. Here we create a non-buffered jobs channel to pass the value of the custom job type. We can also create a buffered done channel with buffer size consistent with the length of the joblist variable []job type (initialization not shown). As these channels and job lists are created, we can start.
These fragments create the first additional co-process. It iterates over the joblist slices and sends each job to the jobs channel. Because the channel is unbuffered, the co-process will immediately block and remain blocked until another process receives data from the jobs channel. Once all jobs have been sent to the jobs channel, the channel will be closed, so the recipient will know that there is no more time for jobs. The semantics of these fragments are not very significant. The For loop runs until it is complete, then closes the jobs channel, but these and other threads in the program occur concurrently. In addition go declares to return immediately, lets the code execute in its own process, of course, at this moment there is no other process to try to explain the jobs, so the co-process is blocked. So, just after the go Declaration, the program has two threads, the main process continues the next declaration, and the most recently created Coprocessor waits for the other to receive data from the jobs channel. Therefore, it takes some time to complete and close the channel before the For Loop. These fragments create a second additional co-process. The Cheng jobs channel, which receives every job, handles the job (which is just print out), and then sends true to the done channel for each job to indicate completion. (We can also send false because we value how many sends are executed in the done channel, not what values are sent.) ） Just like the first go Declaration, this declaration immediately returns, for the declaration of a blockage waiting to be sent. So, at this moment, three concurrent co-processes are executing, the master and two additional co-processes, and 7.3 describe the same. When we have received a send wait (in the # # process), the job is immediately accepted (by the co-thread) and executed. During #1协程再次被堵塞, this time waits for a second job to be sent. Once the # # process has been executed, it is sent to the done channel, which is buffered so it does not clog when it is sent. The control is transferred to the # # of the coprocessor amount for loop, and the next job is sent from the # # coprocessor, and is received by the # # coprocessor, all the way down, knowing that all the work is done.
This is the final fragment, ready to be executed immediately after the other two additional processes have been created and ready to be executed. This code is in the main process, and its purpose is to ensure that the main process is terminated until all work is completed. The for iteration is as many times as jobs, where each iteration is completed from the done channel (discarding the result) to ensure that each iteration is synchronous and complete for each job. If nothing can be received (because a job is being executed but not finished), the reception will be blocked. Once all jobs have been completed, the number of sent and received from the done channel will be the same as the number of iterations, and the For loop will be completed. At this point, the main process will end, so the entire program terminates and we are sure that all the processing is done. Two thumb rules are usually applied to a channel. First, we only need to close the channel when we will check if it is too close (use the For ... range loop, select, or check that the receiver uses the <-action). Second, a channel should be closed by the sender's association, rather than being closed by the receiver's co-process. It is very sensible not to close the channel so that it is never used to check if the channel is closed and the channel is very lightweight, so they do not consume resources, such as opening a file. In this example, according to our thumb rule, the jobs channel iterates through the loop with For...range, so we close it in the sender Association. On the other hand, we don't have to worry about whether the done channel is turned off, so there is no declaration depending on when he is closed. This example shows a go concurrency change into normal mode, although using concurrency in this particular example is not really good. The example of the following module uses a pattern similar to this one, and also fully uses concurrency. 7.2 Examples Although go uses a fairly small amount of syntax to provide co-paths and channels (<-,chan,go,select), this is enough to implement concurrency in a variety of ways. In fact, there are many different ways to do it, and it is impractical to explain each of these changes in this chapter. So, we'll focus on the three-mode, channel, multiple independent concurrency (synchronous and unsynchronized results) that are typically used in concurrency, and a number of interdependent concurrent tasks, and then look at the unique way that each uses the concurrency support for go. Examples are shown here, and in the end there are enough exercises to understand and practice go programming, and these and other methods are safe to use in new programs. 7.2.1 Example: Filter The first example is designed to show a unique concurrency pattern. Programs can easily be adapted to other tasks that benefit from the concurrency of the program. Those who use the UNIX environment may have discovered the Go channel, which is associated with UNIX pipelines (except that the channel is bidirectional and the pipeline is unidirectional). These pipelines can be used to create a channel that passes the output of one program to another, as input to another program, and then returns the output to a third program, and so on. For example, using the UNIX pipeline command find $GOROOT/src-name "*.go" | Grep-v Test.go, we can get a list of all go files included in the Go Resource tree (excluding test files). One of the highlights of this approach is easy to expand. For example, we can add |xargs wc-l to get and list the number of rows that each file contains (at the end, plus the total), and increase the |sort-n to sort by the number of rows (minimum to maximum).
A true Unix-type pipeline is an IO that can be created using the standard library. Pipe () function. For example, the Go standard library uses this function to compare images (see file Go/src/pkg/image/png/reader_test.go). In addition to using IO. Pipe () to create a Unix-type pipeline, you can also use the channel to create a pipeline, Technologies we will review here. The filter example program (in the Filter/filter.go file) receives some command-line arguments (for example, the maximum minimum length of the file and the file suffix that can be received), the list of files and the output, and a list of files that match the given command line. Here the two lines of code are the contents of the main () function in the program. The Handlecommandline () function (not shown) uses the standard library's flag package to handle command-line arguments. The pipe works by calling the outermost (sink ()) from the innermost function (source (file)). Here is the same pipeline, presented in an easy-to-understand way. The source () function, gets a file name slice and returns the Chan type of channel, named Channel1 variable. The source () function alternately sends each file name to the channel. Two filter functions each has a filter standard and a Chan string, each of which returns its own Chan string. In this example, the return channel of the first filter is named Channel2, and the second is channel3. The filter iterates over the items that are received by the channel and sends each item that matches their criteria to the channel that they have returned. The sink () function iterates through the contents of the channel and prints out each one. Figure 7.4 Provides an illustration of how this happens. In this example, the filter program, the sink () function is executed in the main coprocessor, and each pipe function (for example, source (), filtersuffixes (), filtersize () is executed in the respective co-routines). This means that each pipe function call is returned directly and executed quickly to the sink () function. At this point, all the threads are executing concurrently, waiting to be sent or waiting to be received until all the files have been processed.
This function creates a channel that is used to pass the file name. It uses a cache channel because this improves throughput in the test. (We often use the memory consumption to speed up the increase). Once the output channel has been created, we create a recursive iteration file and send each one to the channel. When all the files have been sent, we close the channel. As always, the go declaration returns immediately, so there is a fairly long interval between sending the first item and sending the last item and closing the channel. The second channel is not clogged (at least, the first 1000 files, and less than 1000 files), but if more is sent it will clog up and know one or more to be received from the channel. As we know before, the default channel is bidirectional, but we can force a channel to become unidirectional. Recall the previous section, the Chan<-type type is a send-only type channel, <-chan type is a only receive channel. At the end of the function, the two-way out channel, which is the only receive channel returned, so the file name can only be received. We can, of course, revert to a two-way channel, but this is a better way to express our intentions. After executing the Go declaration, the anonymous function is processed within its own coprocessor, and the function immediately returns to the channel, and the function of the coprocessor sends the name of the file to it. So once the source () function is called, there are two threads executing, the dwell and the other one created in the function.
This is the first of the two filter functions, and only one is shown because the Filtersize function is essentially the same as the structure. The in channel parameter is either a receive channel or a bidirectional channel, but in either case, the type declaration ensures that it can only be received within the filtersuffixes function. (We know that the return value from the source () function is the in channel, which is actually a channel that can only be received.) Accordingly, we return a two-way out channel as a receive-only channel, just as we did in the source () function. In both cases, we omit <-s, and the function works the same way. However, by including the inverse, we have precisely expressed the semantics of what we want to function and make sure that the compiler executes them. The Filtersuffixes function starts with the creation of an output channel with the same size cache as the input channel, resulting in maximum throughput. The function then creates a co-process to handle. In the coprocessor, the in channel is iterated (the file name is received in turn). If no suffix is specified, any suffix can be simply sent to the output channel. If a lowercase suffix is matched to any of the file suffix filenames that can be received, it will be sent to the output channel, otherwise discarded. (filepath. The EXT () function returns the extension of the file name, which is its suffix, including the previous part, or the name of an empty string is not extended) Just like the source () function, once all the processing has ended, the output channel is closed, although it may take some time to reach that point, but the co-process of creating the output channel is returned so that the next function's pipeline can receive the file name. At this point, the three processes are running, the main coprocessor, the source () function's co-process, and the function's covariance. After calling the Filtersize () function, there will be a fourth process, all of which work in parallel. The source function, along with two filter functions, is processed in its own concurrency process, communicating through the channel. The sink () function iterates over the channel operations returned by the last function in the main process, iterating over the file names that were successfully passed from the filter, and outputting them.

The range of the sink () function declares that the iteration receives only the channel, promises the filename or is blocked until the channel is closed, so that the main process is not terminated until the processing of all other threads is complete. Naturally, we add additional functions to the pipeline, either filtering the file names or processing the files that have passed through the filter, when each new function receives an input channel (the output channel of the preceding function) and returns its own output channel. Of course, if we want to pass more complex values through the pipeline, we can also put the channel based on a structure instead of a string. The pipeline shown in this section is a good example of a pipeline framework, and doing specific processing at each stage is really Less benefit from the piping approach. This type of pipeline can benefit from concurrency, and each stage of a pipeline may have a lot of work to do, depending on the project being processed, so that as much time as possible makes the process busy.

"Programming in Go" The seventh chapter concurrent programming translation