[A usage scenario for the Go language]goroutine

This is a creation in Article, where the information may have evolved or changed. Recently, using the go language to write a file conversion tool, the use of goroutine to improve processing efficiency. In the process of writing code, the use of Goroutine experienced three versions of the change, I think it is necessary to record and summarize, but also need to constantly think about how to use the Go language thinking to write the Go language code. Writeyour go code in a go.
Purpose: To traverse a directory and its subdirectories, to convert certain files to another file and save them to a specific directory. Because the file may have many, do not want each file to correspond to a goroutine suddenly have very many goroutine. After all, the CPU cores are fixed quantities. While there are many and many goroutine will not be executed at the same time resulting in a high cost of physical thread switching, but after all, it takes up a lot of memory, and the number is not very predictable. So we are looking for a way to maximize the efficiency of CPU usage while controlling the number of goroutine.
First version: concurrency is a classic producer-consumer model in which a producer passes the traversed file path over a message queue to multiple consumers waiting on this message queue.
Iterate through the specified directory and pass the qualified file path to the callback function collect, which is implemented slightly. Func Walk (path string, Collectfunc (String))//File conversion, implementation slightly. Func convert (file string)
Be processed. Func Main () {

var wg sync. Waitgroup

//Message Queuing

ch: = Make (chan string,runtime. NUMCPU ())

//Create consumer

For

I: = 0; I <runtime. Numcpu (); i++ {

go func () {

For

{

file: = <-ch

convert(file)

WG. Done ()

}

}()

}

//producer

Walk (Path, func (filestring) {

WG. ADD (1)

Ch <-File

})

//wait for all file processing to complete

WG. Wait ()

The problem with the first version is that the processing logic is separated and obviously split into two parts of producer Logic and consumer logic, and my goal is to make the code logic look as serializable as possible, like a sequential program, so it's clearer and easier to understand. Therefore, we need to put the logic of producer and consumer logically together as much as possible.
The second version: Message Queuing, but messages in Message Queuing are not data, but functions. As long as the consumer executes this function, it does not need to care about the implementation of the function and its business logic. Func Main () {

var wgsync. Waitgroup

//Message Queuing

Ch: =make (chan func (), runtime. NUMCPU ())

//Create consumer

For

I: = 0; i< runtime. Numcpu (); i++ {

go func () {

for{

f: =<-ch

f ()

}

}()

}

//producer

Walk (Path,func (file string) {

WG. ADD (1)

ch<-func () {

convert (file)

WG. Done ()

}

})

//wait for all file processing to complete

WG. Wait ()

This version is much better, and the business logic of getting files and processing files together does not lead to a leap of thought. But could it be better? The code of the consumer Goroutine since andBusiness logic is not directly related, is it possible to streamline it?
Third version: The third version of the message queue removed, to a limited number of tokens, to process a file must first obtain a token, the use of the completion of the return. The token is blocked, and if there is no token available for the time being, it will block it, avoiding excessive amounts of uncontrolled goroutine. tokens are implemented in the same way as message queues or with channel, but the meaning is different. Func Main () {

var wgsync. Waitgroup

//Generate a specified number of tokens

tokens: =make (chan int, runtime. NUMCPU ())

Walk (Path, func (file string) {

tokens<-1//Get token WG. ADD (1)

Go func (

) {

convert (file)

WG. Done ()

<-tokens//Return token

}()

})

//wait for all file processing to complete

WG. Wait ()

Now the code is clear and intuitive, and the lines of code are much less.