Golang gc problem (RPM)

This is a creation in Article, where the information may have evolved or changed.

In the actual use of the go language in the process, encountered some seemingly strange memory consumption, and decided to go to the garbage collection model to do some research. This paper summarizes the results of the study.

What is garbage collection?

Once upon a while, memory management was a major challenge for programmers to develop applications. In a traditional system-level programming language (mainly referred to C + +), the programmer must manage the memory carefully and control the memory application and release. A bit of carelessness can create a memory leak, a problem that is difficult to spot and hard to locate, and has been a nightmare for developers. How to solve the problem of this headache? In the past, two approaches were generally used:

• Memory Leak Detection Tool. The principle of this kind of tool is usually static code scanning, through the scanner to detect possible memory leaks of the code snippet. However, the detection tools inevitably have omissions and deficiencies, can only play a supplementary role.
• Smart pointers. This is an automatic memory management method introduced in C + +, which refers to objects by a pointer object that has automatic memory management, which is the purpose for which the programmer does not pay much attention to the release of memory and to the memory for automatic release. This approach is the most widely used approach, but there is a certain amount of learning cost to programmers (not native support at the language level), and memory leaks cannot be avoided once forgotten scenarios are in use.

To solve this problem, almost all the new languages (java,python,php, etc.) that were developed later introduced automatic memory management at the language level – that is, the user of the language only uses memory-focused applications without concern for the release of memory, which is freed by virtual machines Machine) or run time (runtime) for automatic management. The behavior of automatic recycling of memory resources that are no longer used is known as garbage collection.

Common methods of garbage collection

Reference count (reference counting)

This is the simplest kind of garbage collection algorithm, similar to the smart pointers mentioned earlier. A reference count is maintained on each object, and the reference count of the referenced object is automatically reduced by one when the object that references the object is destroyed or updated, and the reference count is automatically added one when the referenced object is created or assigned to another object. Objects are reclaimed immediately when the reference count is 0.

The advantage of this approach is that it is simple to implement, and that memory recovery is timely. This algorithm is widely used in systems with high memory tension and real-time, such as iOS cocoa Framework, Php,python, etc. The simple reference counting algorithm also has obvious drawbacks:

• Frequent update of reference counts reduces performance. A simple workaround is for the compiler to merge adjacent reference count updates into one update, and one way to do this is to not count the frequently occurring temporary variable references, but to decide whether or not to dispose of the temporary object references by scanning the stack when the reference reaches 0. And so there are many other ways, specifically can be referred to here.
• Circular reference issues. Objects in the reference chain cannot be freed when circular references occur between objects. The most obvious solution is to avoid circular references, such as the cocoa introduction of the strong pointer and the weak pointer two pointer types. Or the system detects circular references and actively breaks the loop chain. Of course, this also increases the complexity of garbage collection.

Mark-Clear (Mark and sweep)

The method is divided into two steps, where the tag iterates through all the referenced objects starting from the root variable, marking the objects that can be accessed through the application traversal as "referenced", clearing after the mark is complete, and reclaiming the memory that is not marked (recycling may be accompanied by defragmentation operations). This method solves the lack of reference counting, but there are obvious problems: every start of garbage collection will suspend all the current normal code execution, recycling is greatly reduced system responsiveness! Of course, there are many variants of the mark&sweep algorithm (such as tri-color notation) that optimize the problem.

Generational collection (generation)

After a lot of practical observation, most objects in object-oriented programming languages have very short life cycles. The basic idea of generational collection is to divide the heap into two or more spaces called Generations (generation). Newly created objects are stored in the new generation (young generation) (generally, the Cenozoic size is much smaller than the old age), and as garbage collection repeats, objects with longer lifecycles will be lifted (promotion) into the old age. Therefore, the new generation garbage collection and the old age garbage collection two kinds of different garbage collection methods came into being, respectively, to perform garbage collection for the objects in their space. The new generation of garbage collection is very fast, a few orders of magnitude faster than the old age, even if the new generation of garbage collection is more frequent, the implementation efficiency is still better than the old garbage collection, this is because most objects life cycle is very short, no need to ascend to the old age.

Go's garbage collector

The general use of Go language garbage collection is the classic mark and sweep algorithm.

• Prior to version 1.3, Golang's garbage collection algorithm was very primitive, and then its performance was widely criticized: go runtime under certain conditions (memory exceeds the threshold or regular such as 2min), suspend all Tasks execution, perform mark&sweep operation, Starts execution of all tasks when the operation is complete. In a more memory-intensive scenario, the GO program will have a very noticeable lag when it comes to garbage collection (Stop the World). This delay is simply intolerable in a background service process that requires high response times! Many teams in the production environment practice go language at home and abroad during this period have stepped on the GC pit more or less. The most common way to solve this problem was to control the amount of memory automatically allocated to reduce the GC load as soon as possible, and to handle the scenarios where large and high-frequency allocations of memory were handled by manual memory management.
• The 1.3 release starts with the Go team's continuous improvement and optimization of GC performance, which has become a major concern for every new release of Go release. In version 1.3, go runtime separates the mark and sweep operations, as before, pausing all task execution and starting Mark,mark to restart the suspended task as soon as it is completed, instead allowing the sweep task to be executed in parallel with other tasks in the same way as the normal coprocessor task. If running on a multi-core processor, go attempts to put the GC task on a separate core to run without affecting the execution of the business code as much as possible. The go team's own argument is to reduce the pause time by 50%-70%.
• There are not many performance changes to the GC for version 1.4 (currently the latest stable version). In version 1.4, many of the runtime's code replaced the native C language implementation and adopted the Go Language implementation, and a major change to the GC could be to achieve an accurate GC. C Language implementation in the GC can not get to the memory object information, so can not accurately distinguish between ordinary variables and pointers, only ordinary variables as pointers, if it happens that the ordinary variable point to the space there are other objects, then this object will not be recycled. While the Go language implementation is fully aware of the type information of the object, only the object pointed to by the pointer is traversed during markup, thus avoiding the heap memory waste (resolving about 10-30%) when C is implemented.
• The 1.5 release Go Team has also made a big improvement to the GC (1.4 has been planted with the introduction of write barrier), and the main goal of the official is to reduce latency. Go 1.5 The garbage collector that is being implemented is "non-generational, mobile, concurrent, tri-colored, marked clear garbage collectors." The generational algorithm mentioned above, is a good garbage collection management strategy, but the 1.5 version did not consider the implementation; I guess the reason is that the pace can not be too big, to gradually improve, go official also said in the 1.6 version of GC optimization to consider. The three-color notation described above is also introduced, and the mark operation of this method can be performed incrementally without having to scan the entire memory space each time, which can reduce the stop the world. As you can see, go all the way up to the 1.5 release, Go's garbage collection performance is also improving, but relatively mature garbage collection system (such as Java JVM and JavaScript V8), go need to optimize the path is still very long (but believe the future must be good ~).

Practical experience

The team in the practice of the Go language also encountered the most and most difficult problem is also memory problems (which GC-based), here to meet the problems and experience summarized under, Welcome to exchange discussion.

Go program memory consumption of large issues

This problem was found in our stress testing of backend services, when we simulated a large number of user requests to access back-office services, and the service modules were able to observe a significant increase in memory consumption. However, when the pressure measurement is stopped, the memory footprint does not decrease significantly. It took a long time to locate the problem, the use of gprof and other methods, still found no reason. Finally found that the original is normal ... There are two main reasons for this,

First, the Go garbage collection has a trigger threshold, which will grow with each memory usage becomes larger (such as the initial threshold is 10MB, the next time is 20MB, and the next time it becomes 40MB ...) ), if the GC go is not triggered for a long time, it will be actively triggered once (2min). When memory usage goes up at peak time, it is almost impossible to trigger a GC by threshold unless you continue to request memory, but wait up to 2min for the active GC to start to trigger the GC.

The second reason is that the go language simply tells the system that the memory is not needed to be used when returning the memory to the system, that it can be recycled, that the operating system will take a "procrastination" strategy, that it does not recycle immediately, but that it will not start recycling until the system is in a tight state of memory.

The issue of long GC time

For back-end programs that require user response events, the Stop the world part-time Golang GC is a nightmare. According to the above, the 1.5 version of Go to complete the above improvements should be a lot of GC performance, but all garbage-collected languages are inevitably faced with performance degradation in GC, for this we should try to avoid frequent creation of temporary heap objects (such as &abc{}, new, make, etc.) to reduce the amount of time spent on garbage collection, consider reusing the array cache for temporary objects that require frequent use; many people use CGO methods to manage their own memory and bypass garbage collection, which is not recommended unless the individual is not recommending it (easily causing unpredictable problems). Of course, the circumstances can still be considered, the effect of this trick is still very obvious ~

The problem of goroutine leaks

One of our services needs to handle a lot of long connection requests, when implemented, for each long connection request each open a read and write to the process, all using endless for loop to continuously process the sending and receiving data. When the connection is closed at the remote end, if the two processes are not processed, they will still run and the channel used will not be released ... It is important to note that when you do not use the process, you must close the channel that he relies on and determine whether the channel is closed by the re-process to ensure its exit.