Garbage collection of Golang

This is a creation in Article, where the information may have evolved or changed.

Our company services are in use Golang, millions of per day of UV, over billion of PV has no problem. Later changed the logic, the homepage can show a list, before this list has not done cache processing, one is because the data is small, altogether only more than 1000, there is a small amount. But the homepage added to the entrance after the problem, memory crazy rise, general memory also accounted for 200m+, this is also included in memory cache in the case, this guy, the highest I have seen the use of 20G memory, but the program restarts and can be good. Of course, this is after the understanding, then did not put these two things together.

Say the whole story.

After the go-live memory crazy rise, the beginning will account for about 10G, do not know why, can only kill drop. The program restarts after a short period of time will be maintained around 200M, but after a 4, 5 days will rise up, and the process of memory size has not affected the user's use. Finally can only talk to operations, see the memory is big once kill ...

At that time the memory is so large, it must be a GC problem, do not know where the memory leaks. We use the Beego, I use it encapsulated memory pprof and CPU detection tools of various checks, really did not find anything suspicious. No clue.

Later, chance coincidence to see the Gopher China 2015 conference video, the whole video is placed in the MU class network, we can also go to see. is the industry with Golang of the big guy's speech. Recommend everyone to see:

1. Go language in the Game Project Application report, Dada in the GC is a pioneer in the country, many people study GC is reference to his data.
2. Go language constructs high concurrency Distributed system practice, Golang in 360 applications.
3. Go 1.4 Runtime, this is really big God, rain scar, the whole runtime speech very clear, I this rookie can listen to understand, want to see the principle of memory allocation to see this.

Then, another thing happened, when we big promotion, the line appeared Can't create more than max_prepared_stmt_count statements (current value: 16382) , the database query volume is too large, the DBA to query the number of 30000 is not. The DBA suspects that we didn't release it, but it was written in a ready-made ORM, and went to see the source code, and it was really released.

Last, add cache. This evil, the database is good, there is no such mistake, and the memory is better, no longer disorderly rise. Today, I saw an article in the developer headlines, "Golang GC Inquiry", which is similar to ours. This also shows that our previous memory bizarre crazy rise, is related to the GC mechanism of Golang.

Here's my understanding of the GC. To ensure that the memory in the program is continuous, Golang will request a large chunk of memory (sometimes, just write a Hello, and the world may monitor the memory and may find that it consumes more memory than expected). When a user's program requests more memory than previously requested memory, runtime it makes a GC and doubles the GC threshold. That is, before the GC is over 10M, the next GC is more than 20M. In addition, runtime timed GC is supported. The reason why our memory is rising is that the amount of traffic is too high, the threshold of GC becomes larger and the recovery frequency is lower when the database is accessed. And in terms of recycling, Golang uses a procrastination strategy, and even the freed memory runtime does not immediately return the memory to the system. This causes the memory to fall down, a memory leak illusion.

Golang occurs during GC, the Stop the world whole program pauses, then goes to mark the entire memory can be recycled variables, after marking the execution of the program, the final asynchronous to reclaim memory. Generally this process will reach 20ms. the time it takes to mark recyclable variables depends on the number of temporary variables. the more temporary the number of variables, the longer the scan time.

So the principle of GC optimization now is to declare as few temporary variables as possible:

1. The local variables are reused as much as possible;
2. If there are too many local variables, you can put these variables into a large structure, so that when scanning can only scan a variable, to reclaim the memory that it contains;

Golang currently in the optimization of GC, the overall effect is similar to the big Java, but the stability above, or not. General pressure test the first wave above the effect is very poor.

###### Reference 1. What needs to be noticed in the practice of the Go garbage collection mechanism? -Dada