Speed comparison of Golang Bufio and ioutil reading files (performance test) and analysis of influencing factors

This is a creation in Article, where the information may have evolved or changed.

Objective

There are 4 main ways to read files Golang:

1. Using the Read method with the file itself
2. Using the Read method of the Bufio library
3. ReadAll () using the Io/ioutil library
4. ReadFile () using the Io/ioutil library

As for the first 3 ways of speed comparison, I was first in Golang several read the file way of comparison, but in the blog comment area someone (Study_c) questioned, and provided the test code. According to the test of this code, the result should be

Bufio > ioutil. ReadAll > file comes with read

I have found several problems or factors in the process of running Study_c test code:

• Read () The size of each chunk read has an effect on the result.
• Continuous testing of the same file, from the system cache or SSD cache loading files, the subsequent test results will be accelerated

So the performance test of this paper is based on Study_c code, and try to test the effect of different block size on the result, and increase the ioutil. ReadFile () tests, and randomly generated files to deal with the fairness of cache impact.

Performance testing

Test environment

Cpu:i5-6300hq
Mem:12gb
Dsk:sandisk Extreme PRO SSD 480GB
Os:win 64bit

Test code 1 "randfiles.go", generate 1-500MB file containing random strings

Package Mainimport ("Math/rand" "FMT" "Flag" "StrConv" "Io/ioutil") const Letterbytes = "ABCDEFGHIJKLMNOPQR STUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZ "//https://stackoverflow.com/questions/22892120/    How-to-generate-a-random-string-of-a-fixed-length-in-golangfunc randstringbytes (n int) []byte {b: = make ([]byte, N) For I: = range b {b[i] = Letterbytes[rand. INTN (Len (letterbytes))]} return b}func randfile (path string,filesizemb int) {b:=randstringbytes (FILESIZEMB * 10        24)//Generate 1-500kb size random string BB: = Make ([]byte, FileSizeMB * 1024x768 * 1024x768) for i:=0;i<1024;i++ {//copy 1024 times Copy (Bb[len (b) *i:len (b) * (i+1)],b)}//fmt. Printf ("%s", b) ioutil. WriteFile (path,bb,0666)}func main () {flag. Parse () Filesizemb,err: =strconv. Atoi (flag.    ARG (0))//1-500mb size of file if err! = Nil{panic (Err)} if FileSizeMB > {panic ("too Large FILE,&GT;500MB")} Randfile ("./random1.txt", FileSizeMB) Randfile ("./random2.txt", FileSizeMB) randfile("./random3.txt", FILESIZEMB) Randfile ("./random4.txt", FileSizeMB) fmt. Printf ("Created 4 files, each file size is%d MB.", FileSizeMB)}

Test Code 2 "Speed.go", performance test

Package Mainimport ("FMT" "OS" "Flag" "IO" "io/ioutil" "Bufio" "Time" "StrConv") func read1 (Path St ring,blocksize int) {fi,err: = os. Open (PATH) if err! = nil{Panic (ERR)} defer fi. Close () Block: = Make ([]byte,blocksize) for{N,err: = fi. Read (block) if err! = Nil && Err! = Io. Eof{panic (ERR)} if 0 ==n {break}}}func read2 (path string,blocksize int) {fi,err: = os. Open (PATH) if err! = Nil{panic (ERR)} defer fi. Close () r: = Bufio.  Newreader (FI) Block: = Make ([]byte,blocksize) for{N,err: = R.read (block) if err! = Nil && Err ! = Io. Eof{panic (ERR)} if 0 ==n {break}}}func read3 (path string) {fi,err: = os. Open (PATH) if err! = Nil{panic (ERR)} defer fi. Close () _,err = Ioutil. ReadAll (FI)}func read4 (path string) {_,err: = Ioutil. ReadFile (PATH) if err! = Nil{panic (Err)}}func main () {flag.  Parse () File1: = "./random1.txt" file2: = "./random2.txt"  File3: = "./random3.txt" file4: = "./random4.txt" Blocksize,_: =strconv. Atoi (flag. ARG (0)) var start,end time. Time start = time. Now () Read1 (file1,blocksize) end = time. Now () Fmt. Printf ("File/read () Cost time%v\n", end. Sub (start)) Start = time. Now () read2 (file2,blocksize) end = time. Now () Fmt. Printf ("Bufio/read () Cost time%v\n", end. Sub (start)) Start = time. Now () read3 (file3) end = time. Now () Fmt. Printf ("Ioutil. ReadAll () Cost time%v\n ", end. Sub (start)) Start = time. Now () read4 (file4) end = time. Now () Fmt. Printf ("Ioutil. ReadFile () Cost time%v\n ", end. Sub (start))}

Test results:

Test 1: Block size is 4KB, this is a common size, unexpectedly ioutil. ReadAll () slowest
Test 2: Block size is 1KB, which is the block size used for the test results mentioned in the preface and is consistent with the test results
Test 3: Block size is 32KB, in the case of large chunks, the number of calls to read () Less, Bufio has no advantage, but the first two is far faster than the Ioutil package of two functions
Test 4: Block size is 16 bytes, in case of small block, no cached files plain read results miserable

Impact factors

After consulting the source code of the Golang standard library, there are different results related to the implementation of each method, the biggest factor is the size of the internal buffer, which directly determines the speed of reading:

1. F.read () The underlying implementation is the system call Syscall. Read (), no dig
2. Bufio. Newreader (f) actually calls Newreadersize (F, defaultbufsize), and defaultbufsize=4096, can be used directly with Bufio. Newreadersize (f,32768) to pre-allocate a larger cache, the essence of the cache is make ([]byte, size)
3. Ioutil. ReadAll (f) actually calls ReadAll (r, Bytes. minread), while bytes. MINREAD=512, the essence of caching is bytes. Newbuffer (Make ([]byte, 0, 512), although bytes. Buffer will automatically grow as per the situation, but each reallocation will affect performance
4. Ioutil. ReadFile (path) is called ReadAll (F, n+bytes. Minread), this n depends on the file size, the file is less than 10^9 bytes (0.93GB), n= file size, is newbuffer a slightly larger than the file size of the cache, very generous; This means that files larger than 1G are like Ioutil.readall (f).
5. But why is the full cache of ReadFile not as good as the first two of the chunks read? My guess is that newbuffer packed byte array performance is certainly not as good as a bare-ben character array:

Conclusion

• Bufio is recommended when the size of the block is less than 4KB each time it is read. Newreader (f), greater than 4KB with Bufio.newreadersize (f, Cache size)
• To read reader, the diagram is easy to use Ioutil. ReadAll ()
• Read files once, using Ioutil. ReadFile ()
• Different business scenarios, choose different reading methods