Understanding the memory usage of the Go language

This is a creation in Article, where the information may have evolved or changed.

When many people begin to touch the go language, there is often some confusion as to why a Hello world consumes so much memory. ”。 Understanding Go Lang Memory Usage is a good explanation for this problem. But "Introduction" is "Introduction", more in-depth content I am afraid to the reader to explore. In addition, the article wrote to the end, the author floated, estimated to have caused some public outrage, so he gave himself to mend knives, left a knife, right a knife ...

———— Translation Divider Line ————

Understand the memory usage of the Go language

December 22, 2014, Monday

warm hint : This is only about the Go language memory introduction, as the saying goes nothing ventured, nothing gained, the reader can carry on more in-depth exploration.

Most Go developers will try a simple Hello World program like this:

Package Mainimport ("FMT" "Time") Func main () {FMT. PRINTLN ("Hi") time. Sleep (* time. Second)}

Then they collapsed completely.

f!*%!$%@# 138 g?! This notebook also only has the memory of G!

Virtual Memory vs resident memory

Go manages memory in a way that may not be the way you used to. It retains a chunk of VIRT from the start, and RSS is close to the actual memory usage.

What is the difference between RSS and VIRT?

The VIRT or virtual address space size is the amount of memory that the program maps and can access.

RSS or resident size is the amount of memory actually used.

If you are interested in how Go is implemented, take a look at this:

Https://github.com/golang/go/blob/master/src/runtime/malloc1.go

    In a 64-bit device, it is allocated from a single contiguous reserved address.    //current (MAXMEM) should be sufficient.    //actually we have reserved 136 GB (because the final bit mapping will use 8 GB)

It is important to note that if you use a 32-bit schema, the memory retention mechanism is completely different.

Garbage collection

Now that we know the difference between resident memory and shared memory, we can talk about the mechanism of Go garbage collection to see how our programs work.

Imagine that you are writing a long-running background service, and let it be a Web application service or something more complex. In general, memory is allocated throughout the operating cycle. It is necessary to know how to handle these memory.

Typically, garbage collection is performed every 2 minutes. If a fragment is not used for 5 minutes, the collector releases it.

So if you think memory usage is going down, then check it out in 7 minutes.

It is important to note that the current GC is uncompressed, meaning that if you have a byte in use on a page, the collector refuses to release the page.

Finally, and most importantly, the goroutine stack of Go 1.3 has 8k/pop space that will not be freed, and they will then be reused. Don't worry, Go has a lot of room for improvement in the GC section. So, if your code generates a lot of goroutine, and RES stays high, that might be the reason.

Well, now we know what the program looks like externally and what it expects the GC to do.

Analyzing Memory usage

Now take a look at how to understand memory usage with a small example. In this example, we will allocate 10 groups of 100 megabytes of memory.

Then there are a number of ways to understand the use of memory.

One method is to pass the Readmemstats function of the runtime package.

Another method is to provide the Web interface through the PPROF package. This allows us to remotely obtain PPROF data for the program, which will be explained in more detail later.

Another way we have to introduce is Dave Cheney, which uses Gctrace to debug environment variables.

Note: These are all done under the GO 1.4 environment under 64-bit Linux.

package mainimport ("Log" "Net/http" _ "net/h Ttp/pprof "" Runtime "" Sync ") func bigbytes () *[]byte {s: = make ([]byte, 100000000) return & ; S}func Main () {var wg sync. Waitgroup go func () {log. Println (http. Listenandserve ("localhost:6060", Nil)} () var mem runtime. Memstats Runtime. Readmemstats (&MEM) log. Println (Mem. Alloc) log. Println (Mem. Totalalloc) log. Println (Mem. HeapAlloc) log. Println (Mem. Heapsys) for I: = 0; I < 10; i++ {s: = Bigbytes () if s = = nil {log. Println ("Oh Noes")}} runtime. Readmemstats (&MEM) log. Println (Mem. Alloc) log. Println (Mem. Totalalloc) log. Println (Mem. HeapAlloc) log. Println (Mem. HEAPSYS) WG. ADD (1) WG. Wait ()} 

When you use Pprof to view memory, you typically use two options.

An option is "-alloc_space" to tell you how much memory has been allocated.

The other is "-inuse_space", which is used to obtain the amount of memory being used.

You can run Pprof and point it to our built-in Web service for maximum memory consumption.

You can also use the list to understand where these memory is used:

Use

vagrant@vagrant-ubuntu-raring-64:~/blahdo$ Go tool pprof-inuse_spaceblahdo http://localhost:6060/debug/pprof/ Heapfetching profile from http://localhost:6060/debug/pprof/heapSaved profile in/home/vagrant/pprof/ Pprof.blahdo.localhost:6060.inuse_objects.inuse_space.025.pb.gzentering interactive mode (type ' help ' for commands) ( PPROF) TOP5190.75MB of 191.25MB Total (99.74%) Dropped 3 nodes (cum <= 0.96MB)      flat  flat%   sum%        cum   cum%  190.75MB 99.74% 99.74%   190.75MB 99.74%  main.main         0     0% 99.74%   190.75MB 99.74%  runtime.goexit         0     0% 99.74%   190.75MB 99.74%  runtime.main (pprof) quit

Distribution

vagrant@vagrant-ubuntu-raring-64:~/blahdo$ Go tool pprof-alloc_spaceblahdo http://localhost:6060/debug/pprof/ Heapfetching profile from http://localhost:6060/debug/pprof/heapSaved profile in/home/vagrant/pprof/ Pprof.blahdo.localhost:6060.alloc_objects.alloc_space.027.pb.gzentering interactive mode (type ' help ' for commands) ( PPROF) TOP5572.25MB of 572.75MB Total (99.91%) Dropped 3 nodes (cum <= 2.86MB)      flat  flat%   sum%        cum   cum%  572.25MB 99.91% 99.91%   572.25MB 99.91%  main.main         0     0% 99.91%   572.25MB 99.91%  runtime.goexit         0     0% 99.91%   572.25MB 99.91%  runtime.main

The leaderboard is pretty good, but the list command is better, and you can see in the context how consumption affects other parts of the program.

(pprof) listtotal:572.75mbroutine ======================== main.main in/home/vagrant/blahdo/main.go  572.25MB   572.25MB (flat, cum) 99.91% of total         .     At:   var mem runtime. Memstats         .     :   runtime. Readmemstats (&MEM)         :     :   log. Println (Mem. Alloc)         ..     :.          .     : For   I: = 0; i < i++ {  572.25MB   572.25MB     :           s: = Bigbytes ()     . :           if s = = nil {         .          .     :                   log. Println ("Oh Noes")         .     :           }         .     :   }         .     33:

Smart readers may have found some differences in the memory usage reports above. Why is that?

Let's take a look at the process:

vagrant@vagrant-ubuntu-raring-64:~$ PS aux | grep blahdovagrant   4817  0.2 10.7 699732 330524 pts/1   sl+  00:13   0:00./blahdo

Now let's look at the log output:

./vagrant@vagrant-ubuntu-raring-64:~/blahdo$./BLAHDO2014/12/23 00:19:37 2796722014/12/23 00:19:37 3361522014/12/23 00:19:37 2796722014/12/23 00:19:37 8192002014/12/23 00:19:37 3002099202014/12/23 00:19:37 10004209682014/12/23 00:19:37 3002099202014/12/23 00:19:37 500776960

Finally, take a look at the effect of using gctrace:

vagrant@vagrant-ubuntu-raring-64:~/blahdo$ godebug=gctrace=1./BLAHDOGC1 (1): 1+0+95+0 us, 0, 0 MB, (21-0) objects  , 2 goroutines, 15/0/0 sweeps, 0 (0) handoff, 0 (0) steal, 0/0/0 yieldsgc2 (1): 0+0+81+0 us, 0--0 MB, (53-1) objects, 3 Goroutines, 20/0/0 sweeps, 0 (0) handoff, 0 (0) steal, 0/0/0 yieldsgc3 (1): 0+0+77+0 us, 0, 0 MB, 151 (169-18) object S, 4 goroutines, 25/0/0 sweeps, 0 (0) handoff, 0 (0) steal, 0/0/0 yieldsgc4 (1): 0+0+110+0 us, 0, 0 MB, 325 (393-68) obj  ECTS, 4 goroutines, 33/0/0 sweeps, 0 (0) handoff, 0 (0) steal, 0/0/0 yieldsgc5 (1): 0+0+138+0 us, 0, 0 MB, 351 (458-107)  Objects, 4 goroutines, 40/0/0 sweeps, 0 (0) handoff, 0 (0) steal, 0/0/0 yields2014/12/23 02:27:14 2779602014/12/23 02:27:14 3326802014/12/23 02:27:14 2779602014/12/23 02:27:14 884736gc6 (1): 1+0+181+0 us, 0-up MB, 599 (757-158) objects, 6 g Oroutines, 52/0/0 sweeps, 0 (0) handoff, 0 (0) steal, 0/0/0 Yieldsgc7 (1): 1+0+454+19 us, 286 MB, 438 (759-321) Obje CTS, 6 Goroutines, 52/0/0 Sweeps, 0 (0) handoff, 0 (0) steal, 0/0/0 Yieldsgc8 (1): 1+0+167+0 us, 477 MB, (762-322) objects, 6 Goroutines , 54/1/0 sweeps, 0 (0) handoff, 0 (0) steal, 0/0/0 yieldsgc9 (1): 2+0+191+0 us, 477 MB, (765-325) objects, 6 go Routines, 54/1/0 sweeps, 0 (0) handoff, 0 (0) steal, 0/0/0 yields2014/12/23 02:27:14 3002068642014/12/23 02:27:14 1000417040 2014/12/23 02:27:14 3002068642014/12/23 02:27:14 500842496GC forcedgc10 (1): 3+0+1120+22 us, 286 MB, 455 (789-334 ) objects, 6 goroutines, 54/31/0 sweeps, 0 (0) handoff, 0 (0) steal, 0/0/0 yieldsscvg0:inuse:96, idle:381, sys:477, Rele ased:0, consumed:477 (MB) GC forcedgc11 (1): 2+0+270+0 us, up-and-down MB, 438 (789-351) objects, 6 goroutines, 54/39/0 SW Eeps, 0 (0) handoff, 0 (0) steal, 0/0/0 yieldsscvg1:0 MB releasedscvg1:inuse:96, idle:381, sys:477, released:0, consum ed:477 (MB) GC FORCEDGC12 (1): 85+0+353+1 us, up to 6 MB, 438 (789-351) objects, goroutines, 54/37/0 sweeps, 0 (0) Han Doff, 0 (0) steal, 0/0/0Yields 

Since most operations tools are the operating system's point of view to treat your program, it's important to understand what's actually going on inside the program.

For more options, refer to the runtime package.

Excerpts are as follows:

• res– will display the memory usage of the process at the current moment, but may not contain any pages that have not been swapped in or have been swapped out.
• Mem. alloc– number of bytes that have been provisioned and are still in use
• Mem. totalalloc– The total amount of memory allocated from the beginning of the run to the present
• Mem. Current dosage of heapalloc– heap
• Mem. heapsys– contains the current amount of the heap and has been released but not yet returned to the operating system

Further, it is important to understand that PPROF is simply acquiring a sample, not a real value.

Usually when dealing with this situation, do not focus on the number itself, but focus on solving the problem.

We believe in measuring everything, but at the same time feel that the "modern" OPS tools are pretty bad and focus on the impact of the problem, not the real problem.

If your car doesn't start, you might think it's a problem, but it's not. This is not even the empty image of the mailbox. The real problem is that you don't refuel your mailbox, but your focus is on a series of results from the initial problem.

If you focus only on the RES value from PS for the Go program, it may tell you that there is a problem here, and that there is no clue to solve it unless you dig further. We would like to be able to correct it.

Correct:

The last paragraph was not edited. It is not intended to degrade OPS or devops personnel. The goal is to demonstrate application-level metrics and system-level metrics. We are aware that the expression here is wrong and we apologize for it. We just feel that the "ops" tool does not provide developers with sufficient information to fix their problems.

We also believe that the current application-level measurement tools are still scarce.

OPS people play a vital role in our sincere thanks for their work. In fact, it's the developer's bad code that's putting things into trouble, and that's what we're working on.

Final corrections

Instead of letting ops have more than 300 charts including tables, counters, point plots, and histograms. As a software writer, we should pay more attention to finding real problems and proposing real solutions.