Golang concurrency, Jane's Way

This is a creation in Article, where the information may have evolved or changed.

Goroutines

• Models: Functions that execute concurrently with other goroutine in the shared address space
• Resource consumption: Initial very small stack overhead, followed by increased or decreased memory on the heap as required
• Create and destroy: The Go keyword means to create a new goroutine (note that it is not executed immediately, but is placed in the scheduled queue for scheduling), and after the function runs, Goroutine automatically destroys

Goroutine is the advantage of Golang, a simple, lightweight concurrency model.

Channel

• One of the data types, similar to Message Queuing, facilitates communication between different goroutine.
• Can be single-channel, may contain various types of data, or can be divided into buffer and without buffer
• Reading data from an empty channel is blocked (the closed pipe will not block), and writing data to a full channel will also block
• Unlike a file, a network socket, close does not release resources, it simply no longer receives more messages, so it does not need to release the channel resources through close, but if you have a range loop, you need to close, or range Loop will block.
• If the data is written to the closed channel, it will be panic, if the data is read, the excess data in the pipeline will be read first, then the 0 value is obtained.
• If you do not know the number of channel, you can use reflect. Select to choose

Sync Package

• Read/write lock, writing lock, Atomic,waitgroup

There ' s a disconnect between the concurrency primitives that Go, and the expectations of the "who try it."
Golang provides a very simple concurrency model, but concurrent programming is still not easy.

Body

Start with the ' go ' keyword first.

package mainimport ( "fmt" "log" "net/http")func main() { http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {     fmt.Fprintln(w, "Hello, GopherCon SG") }) go func() { if err := http.ListenAndServe(":8080", nil); err != nil {     log.Fatal(err) } }() for {}}hello world web2.0 版本

What's wrong with the above code?

for {}

for{}is a dead loop that will always occupy the CPU and cause the CPU to spin.
How to solve it?

for { runtime.Gosched() }

Give up the CPU, but this approach still consumes CPU and does not solve the underlying problem. There is a better way, with an select{} alternative for{} , empty select{} statements will always block.
The above example simply demonstrates some minor problems that are not formally used, and the following is the correct example that we often use:

package mainimport ( "fmt" "log" "net/http")func main() { http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {     fmt.Fprintln(w, "Hello, GopherCon SG") }) if err := http.ListenAndServe(":8080", nil); err != nil {     log.Fatal(err) }}

If you have a to-wait for the result of an operation, it's easier to do it yourself.

• The first suggestion: if you need to wait for the result of an operation, you do not need to create a new goroutine to run this operation, while blocking the outer goroutine

Since the biggest feature is concurrency, of course you can't miss the concurrency example:

func restore(repos []string) error {     errChan := make(chan error, 1)     sem := make(chan int, 4) // four jobs at once     var wg sync.WaitGroup     wg.Add(len(repos))     for _, repo := range repos {         sem <- 1         go func() {         defer func() {             wg.Done()             <-sem         }()         if err := fetch(repo); err != nil {            errChan <- err        }     }()     }     wg.Wait()     close(sem)     close(errChan)     return <-errChan}这个是gb-vendor早期的一个版本,并发的获取依赖资源

Look carefully, think of the code, what can only be the problem?
First look at this pair of code blocks:

defer func() {  wg.Done()  <-sem}()和这段:wg.Wait()close(sem)

Close (SEM) in WG. Wait () after the WG. Wait () in WG. Done (), but is not guaranteed to occur after <-sem, that is, close (SEM) and <-sem who first who is not guaranteed. So is it possible to cause panic?
Refer to the above introduction to the channel: reading data from a closed channel, (if any) first taking out the data in the pipeline, and then returning the 0 value directly, without blocking.
Simply modify the next defer to make the order of execution clearer:

func restore(repos []string) error {     errChan := make(chan error, 1)     sem := make(chan int, 4) // four jobs at once     var wg sync.WaitGroup     wg.Add(len(repos))     for _, repo := range repos {         sem <- 1         go func() {             defer wg.Done()             if err := fetch(repo); err != nil {                 errChan <- err             }         <-sem     }()     }     wg.Wait()     close(sem)     close(errChan)     return <-errChan}

• Second recommendation: The order in which locks and semaphores are released is reversed from the order in which they are acquired.
  There is a type of multi-layer lock, the inner lock is released first, then the outer layer.

Why Close (SEM)?

Unlike a file, a network socket, close does not release resources, it simply no longer receives more messages, so it does not need to release the channel resources through close, but if you have a range loop, you need to close, or range Loop will block.
There is no range loop for the channel, so you can delete the close (SEM)

Let's look at how SEM is used.

SEM is designed to run at any time with only a limited fetch operation. Look closely at the previous code, what is the question?
The code only guarantees that no more than 4 Goroutine are running, not 4 fetch operations are running, and then the next one is not.
The preceding code only guarantees that no more than 4 Goroutine will run, and when the fifth repo, it blocks the for loop, waiting for a previous goroutine to execute and then create a new Goroutine (which will not be executed immediately), relatively inefficient.
Another is to put the required goroutine into the scheduling pool and then run directly:

func restore(repos []string) error {     errChan := make(chan error, 1)     sem := make(chan int, 4) // four jobs at once     var wg sync.WaitGroup     wg.Add(len(repos))     for _, repo := range repos {         go func() {                              defer wg.Done()             sem <- 1             if err := fetch(repo); err != nil {                 errChan <- err             }         <-sem     }()     }     wg.Wait()     close(errChan)     return <-errChan}

The SEM <-1 is put into go func inside, all the goroutine will be created and executed immediately.

• Recommendation three: For semaphores, where to use is where to get

The bugs are all done?

Back to the code above, note for: Range and Fetch (repo) code block, you see what the problem?
There are two problems:
Variable repo in 1.goroutine will change with each iteration, which may cause all fetch operations to fetch the last value
2. If the variable repo has both read and write operations, it will cause competition

How to deal with it? To add parameters to an anonymous method:

func restore(repos []string) error {     errChan := make(chan error, 1)     sem := make(chan int, 4) // four jobs at once     var wg sync.WaitGroup     wg.Add(len(repos))     for i := range repos {         go func(repo string ) {                              defer wg.Done()             sem <- 1             if err := fetch(repo); err != nil {                 errChan <- err             }         <-sem     }(repos[i])     }     wg.Wait()     close(errChan)     return <-errChan}

• Recommendation 4: Avoid using external variables directly in goroutine, preferably as arguments

The last one, huh?
Wait, one more bug

Return to the above code, carefully observe the processing of Errchan and fetch error, it is estimated that the death can not see the problem?
Give a little hint, if more than one error, what will happen?
Close (Errchan) relies on the WG. Wait () executes first, WG. Wait () relies on the WG. Done () performed first, WG. Done is dependent on Errchan <-err first execution, but the Errchan buffer only 1,goroutine there are four. But when more than one error occurs, the Boom...,errchan <-err operation is blocked, forming a deadlock.
Workaround, the Errchan buffer equals the number of repos:errChan := make(chan error, len(repos))

• The last piece of advice: when you create a goroutine, you need to know when and how to quit this Goroutine

The concurrency model provided by Golang is simple, but with good concurrency you need to master a variety of common patterns and scenarios, not just language knowledge

Personal blog: blog.duomila.club

Resources:
Https://dave.cheney.net/paste/concurrency-made-easy.pdf