Golang Concurrent downloads of multiple files

Background Notes

Suppose you have a distributed file system, and now you need to download a subset of files from that system to the local machine.
A partial IP of the file system is known, as well as a list of Fileid files that need to be downloaded, and this information can be spliced to the download address.
Where the node IP list is saved in Xx_node.txt, the FileID to be downloaded is saved in Xx_fileid.txt. code Example

Package main import ("Bufio" "Flag" "FMT" "io" "Math/rand" "Net/http" "OS" "Time") var (clustername = flag.) String ("ClusterName", "C1", "Download ClusterName"))//Line read file contents func readlines (Fpath string) []string {fd, err: = Os. Open (Fpath) if err! = Nil {panic (err)} defer FD. Close () var lines []string Scanner: = Bufio. Newscanner (FD) for scanner. Scan () {lines = append (lines, scanner. Text ())} If Err: = Scanner. ERR (); Err! = Nil {fmt. Fprintln (OS. Stderr, err)} return lines}//implement download of single file Func Download (ClusterName String, node string, FileID String) string {nt: = time. Now (). Format ("2006-01-02 15:04:05") fmt. Printf ("[%S]TO download%s\n", NT, FileID) URL: = fmt. Sprintf ("http://%s/file/%s", node, FileID) Fpath: = Fmt. Sprintf ("/yourpath/download/%s_%s", ClusterName, FileID) newFile, err: = OS. Create (Fpath) if err! = Nil {fmt. Println (Err. Error () return "process failed for" + FileID} defer Newfile.close () Client: = http. client{timeout:900 * time. Second} resp, err: = client. Get (URL) defer resp. Body.close () _, err = Io. Copy (NewFile, resp. Body) If err! = Nil {fmt. Println (Err. Error ())} return FileID} func main () {flag. Parse ()//reads the node IP list from the file nodelist: = ReadLines (FMT. Sprintf ("%s_node.txt", *clustername)) If Len (nodelist) = = 0 {return}//Read the list of file IDs to be downloaded from the file FileID List: = ReadLines (FMT.

    Sprintf ("%s_fileid.txt", *clustername)) If Len (fileidlist) = = 0 {return} ch: = Make (Chan string) Each goroutine handles the download of a file R: = Rand. New (Rand. Newsource (time. Now ().  Unixnano ()))) for _, FileID: = Range fileidlist {node: = NODELIST[R.INTN (len (nodelist))] Go func (node, FileID string) {ch <-Download (*clustername, node, FileID)} (node, FileID)}//wait for each file to be downloaded To complete and check the timeout timeOut: = time. After (*. Second) for idx: = 0; IDX < Len (fileidlist); idx++ {Select {case res: = <-ch:nt: = time. Now (). Format ("2006-01-02 15:04:05") fmt. Printf ("[%s]finish download%s\n", NT, RES) case <-timeout:fmt. Println ("Timeout ...") Break}}}

Summary

The default HTTP Client is not used for the download and specifies the time-out period;
The system call is called when the file is downloaded, goroutine will be suspended;
When the downloaded file is complete, the suspended Goroutine is awakened, and the goroutine exits after executing the following code;
Global timeout control, timeout after the main thread exits.