[Golang] Memory growing Bytes.makeslice

This is a creation in Article, where the information may have evolved or changed.

Golang write a picture server, in the bulk download compression time to find the memory growing ....

Fortunately, the Golang comes with a memory footprint log combined with an analysis tool to easily see the memory distribution.

For details, refer to:

Http://blog.golang.org/profiling-go-programs

Cpu\ memory information can be counted in real time.

The main point here is how the memory is engaged. A reference to the CPU analysis prior to an article.

//need to include this PPROF packageImport  "runtime/pprof"//here receive memory statistics save file var memprofile= Flag. String ("Memprofile","","Write memory profile to the This file")//here is the logic to determine if memory needs to be recordedif*memprofile! =""{var err error memfile, err = OS. Create (*memprofile)ifErr! =Nil {Log.Println (ERR)}Else {            Log. Println ("Start write heap profile ....") Pprof.Writeheapprofile (memfile) defer memfile.Close()        }    }Here is a more flexible way to turn on logging and closing records as HTTP requests, when needed to open \ When not needed to shut down. Remember to add token

The full code is as follows:

//godebug=schedtrace=1000./trace_example//gomaxprocs=2 godebug=schedtrace=1000./trace_example//gomaxprocs=2 godebug=schedtrace=1000,scheddetail=1./trace_example PackageMainImport (    "Flag" "Log" "OS" "Runtime/pprof"//"Net/http"//_ "Net/http/pprof""Sync" "Time")//http://www.graphviz.org/Download_macos.php//var cpuprofile = flag. String ("Cpuprofile", "", "Write CPU profile to file")var memprofile = flag. String ("Memprofile", "", "write memory profile to this file") var memfile*OS. Filefunc Main () {flag. Parse ()//if *cpuprofile! = "" {//F, err: = OS. Create (*cpuprofile)//If Err! = Nil {//log. Fatal (ERR)//     }    //pprof. Startcpuprofile (f)//defer pprof. Stopcpuprofile ()// }    if*memprofile! = ""{var err error memfile, err= OS. Create (*memprofile)ifErr! =Nil {log. PRINTLN (ERR)}Else{log. Println ("Start write heap profile ...") pprof. Writeheapprofile (memfile) defer Memfile.close ()}}//go func () {//log. Println (http. Listenandserve ("localhost:6060", nil))// }()var wg sync. Waitgroup WG. ADD (10)     forI: = 0; I < 10; i++{go work (&WG)} Wg. Wait ()//Wait to see the Global Run queue deplete.Time. Sleep (300 *Time . Second)}func Work (WG*sync. Waitgroup) {time. Sleep (time. Second) var counterint     forI: = 0; i < 1e10; i++{time. Sleep (Time.millisecond* 100) pprof. Writeheapprofile (memfile) counter++} WG. Done ()}

OK, plus this memory analysis data, continue to run the service, after running for some time, stop the program, using the following command to analyze.

Go Tool pprof Image_service  

(pprof) Top202622.12MB of 4938.25MB Total (53.1%) DroppedNodes (cum <= 24. 69MB) Showing TopNodes out of (cum >= 419. 23MB) Flat flat% sum% cum cum%1759.43MB 35.63% 35.63% 1759.43MB 35.63% bytes.makeslice203.06MB 4.11% 39.74% 320.58MB 6.49% net/Url.parsequery166.11MB 3.36% 43.10% 166.11MB 3.36% Net/textproto. (*Reader). ReadLine132.03MB 2.67% 45.78% 132.03MB 2.67% Net/textproto. (*Reader). Readmimeheader117.52MB 2.38% 48.16% 117.52MB 2.38% net/Url.unescape71.02MB 1.44% 49.60% 71.02MB 1.44%Mcommoninit60.50MB 1.23% 50.82% 60.50MB 1.23%FMT. Sprintf37.51MB 0.76% 51.58% 98.01MB 1.98% _/home/qingpingzhang/project/createdji_servers/image_service/image. (*Handler). Handleredo35.51MB 0.72% 52.30% 333.65MB 6.76% net/http. Readrequest21.37MB 0.43% 52.73% 21.37MB 0.43% github.com/gographics/imagick/imagick._cfunc_gobytes17.57MB 0.36% 53.09% 17.57MB 0.36%Bufio. Newreadersize0.50MB 0.01% 53.10% 21.58MB 0.44% net/http. (*Transport). Dialconn0 0% 53.10% 21.87MB 0.44% _/home/qingpingzhang/project/createdji_servers/image_service/image. (*Handler). Compresswithsizelist0 0% 53.10% 1781.66MB 36.08% _/home/qingpingzhang/project/createdji_servers/image_service/image. (*handler). Dorecompress 0 0% 53.10% 1759.29MB 35.63% _/home/qingpingzhang/project/createdji_servers/image_service/image . (*handler). Httpgettomagickwand0 0% 53.10% 17.57MB 0.36%Bufio. Newreader0 0% 53.10% 1759.43MB 35.63% bytes. (*buffer). Readfrom0 0% 53.10% 21.37MB 0.43% Github.com/gographics/imagick/imagick. (*Magickwand). Getimageblob0 0% 53.10% 419.23MB 8.49% Main. (*imageservice). Servehttp0 0% 53.10% 419.23MB 8.49%Main. Action (pprof) Quit

Initially you can navigate to when download compression is assigned too much byteslice caused.

Observe the code and find no specific reason until the article was found online:

http://openmymind.net/Go-Slices-And-The-Case-Of-The-Missing-Memory/

Buffer: = bytes. Newbuffer (make ([]byte, 0, resp. contentlength) buffer. Readfrom (Res. Body) Body:= buffer. Bytes ()

A Memory Leak

Look, what's a memory leak within the context of a runtime that provides garbage collection? Typically it ' s either a rooted object, or a reference from a rooted object, which you haven ' t considered. This is obviously different as it's really extra memory you might isn't be aware of. rooting the object might very well being intentional, but you don't realize just how much memory it's you ve rooted. Sure, my ignorance is at least 75% to blame. Yet I can ' t help but shake the feeling that there's something too subtle about all of this. Any code can return something this looks and quacks like an array of 2 integers yet takes gigs of memory. Furthermore, as bytes.MinRead a global variable is just bad design. I can ' t imagine how many people think they ' ve allocated X when they ' ve really allocated x*2+512.

The general meaning is that this buffer uses the smallest unit to read, if not enough, then continue to apply for twice times the size of space.

Can view the source code:

146//Readfrom reads data from R until EOF and appends it to the buffer, growing147//The buffer as needed. The return value n is the number of bytes read. any148//error except IO. EOF encountered during the read is also returned. If the149//buffer becomes too large, readfrom'll panic with Errtoolarge.Max Func (b *Buffer) Readfrom (R io. Reader) (n Int64, err error) {151 B.lastread =Opinvalid152//If buffer is empty and reset to recover space.153ifB.off >=Len (b.buf) {154 B.truncate (0)   155        }   156 for {   157ifFree: = Cap (B.BUF)-Len (B.BUF); Free <Minread {158//Not enough space at end159 Newbuf: =B.buf160ifB.off+free <Minread {161//Not enough space using beginning of buffer;162//Double buffer capacity163newbuf = Makeslice (2*cap (b.buf) + minread)164                }   165copy (Newbuf, B.buf[b.off:])166 B.buf = Newbuf[:len (b.buf)-B.off]167 B.off = 0 168            }   169 m, E: =R.read (B.buf[len (b.buf): Cap (B.BUF)])B.buf = B.buf[0:len (b.buf) +m]171 N + =Int64 (m)172ifE = =io. EOF {173 Break174            }   175ifE! =Nil {176returnN, E177            }   178        }   179returnN, Nil//err is EOF, so return nil explicitly180}

Solution:

//Ioutil. ReadAll starts at a very small//It really should let's specify an initial sizeBuffer: = bytes. Newbuffer (make ([]byte, 0, 65536) ) Io. Copy (buffer, r.body) Temp:=buffer. Bytes () Length:=Len (temp) var body []byte//Is we wasting more than 10% space?ifCap (temp) > (length + LENGTH/10) {Body= Make ([]byte, length) copy (body, temp)}Else{Body=Temp}

After a bit of testing, the memory is garbage collected. Why is there such a situation?

Entering Interactive mode (type "Help" forcommands) (pprof) Top20834.66MB of 1599.63MB Total (52.18%) Dropped175 nodes (Cum <=8MB) Showing TopNodes out of (cum >= 72. 01MB) Flat flat% sum% cum cum% 427.45MB 26.72% 26.72% 427.45MB 26.72%Bytes.makeslice185.80MB 11.62% 38.34% 614.25MB 38.40% _/home/qingpingzhang/project/createdji_servers/image_service/image. (*Handler). Httpgettomagickwand69.01MB 4.31% 42.65% 69.01MB 4.31% Net/textproto. (*Reader). Readmimeheader 48MB3.00% 45.65% 48MB 3.00% net/Url.unescape24.51MB 1.53% 47.18% 24.51MB 1.53%Mcommoninit24.01MB 1.50% 48.68% 72.01MB 4.50% net/url.parsequery 24MB1.50% 50.19% 117.02MB 7.32% net/http. Readrequest 24MB1.50% 51.69% 24MB 1.50% net/Url.parse7.87MB 0.49% 52.18% 7.87MB 0.49% github.com/gographics/imagick/imagick._cfunc_gobytes0 0% 52.18% 7.87MB 0.49% _/home/qingpingzhang/project/createdji_servers/image_service/image. (*Handler). Compresswithsizelist0 0% 52.18% 622.62MB 38.92% _/home/qingpingzhang/project/createdji_servers/image_service/image. (*handler). Dorecompress 0 0% 52.18% 427.95MB 26.75% bytes. (*buffer). Readfrom0 0% 52.18% 7.87MB 0.49% Github.com/gographics/imagick/imagick. (*Magickwand). Getimageblob0 0% 52.18% 72.01MB 4.50% Main. (*imageservice). Servehttp0 0% 52.18% 72.01MB 4.5%Main. Action0 0% 52.18% 72.01MB 4.50% net/http. (*Request). Parseform0 0% 52.18% 117.02MB 7.32% net/http. (*conn). Readrequest0 0% 52.18% 117.02MB 7.32% net/http. (*conn). Serve0 0% 52.18% 72.01MB 4.50% net/http.func 014 0 0% 52.18% 72.01MB 4.50% net/url. Parsequery

In the Golang language comes with the bytes pack inside the application memory, why will not quickly be recycled?

Puzzled, IO operation this piece also need to find time to re-learn.