Linux Debugging (ix) "Memory leak" in a production environment

An accidental opportunity to discover that a process uses more than 14G of memory. This process is an RPC server, just as a transit, absolutely should not use so much memory. Even if the amount of concurrency is too much. There is too much data in memory. In the case of concurrency reduction, this memory usage will definitely fall down. But actually. This memory will always rise. Until it was killed by Oom killer.

Because this RPC server logic is relatively simple. First the daytime source code, in addition to find some simple programming above the problem. There's no big problem. First on Valgrind:

Valgrind--tool=memcheck--leak-check=full-v./rpc_server

The original case is that the memory leaks are generally checked out. Not this time (otherwise there will be no article):

In fact at least 10G of "memory leaks". Since it's not checked out, it means that the memory is still alive. Imagine this scenario: each request is a new piece of memory, placed in a list. After the normal request is processed, you need to remove the memory from this list. Assumptions are not removed. Then this is a memory leak. But Valgrind can't check it out.

Because the above process uses the Tcmalloc, is not the tcmalloc problem? We know that tcmalloc is more efficient than malloc. So is not the tcmalloc problem, assuming that it has been applying for memory, not released, will cause such a "memory leak."

Note the following passage:

Releasing memory back to the Systemby default, Tcmalloc would release no-longer-used Memory back to the kernel gradually, O Ver time. The TCMALLOC_RELEASE_RATE flag controls how quickly this happens. You can also force a release at a given point in the progam execution like so:   mallocextension::instance ()->release Freememory (); can also call Setmemoryreleaserate () to change the Tcmalloc_release_rate value at runtime, or getmemoryreleaserate to See what's the current release rate is.

A simple translation is a mechanism for tcmalloc memory back into the OS: By default, Tcmalloc will return unused memory for a long time to the system. Tcmalloc_release_rate This flag controls the frequency of this turn-back. You can force this release to occur at execution time by this statement:

Mallocextension::instance ()->releasefreememory ();

Of course. You can pass SetMemoryReleaseRate() To set this tcmalloc_release_rate. The assumption is set to 0. The representative never returned. The larger the number represents the greater the frequency of the return.

The general reasonable value is to set a number between 0-10. You can also set this rate by setting the tcmalloc_release_rate for the environment variable.

With this suspicion, first check the heap usage using Google's gpreftools:

1. Export Heapcheck=draconian

2. Export Pprof_path=/usr/local/bin/pprof

You can start it directly.

The reason is draconian set to. Because of the need for more specific statistical information. More will believe explanations such as the following: flavors ofHeap Checking

These is the legal values when running a whole-program heap check:

1. minimal
2. normal
3. strict
4. draconian

" Minimal "heap-checking starts as late as possible in a initialization, meaning can leak some memor Y in your initialization routines (that run before   main () , say), and not trigger a leak message. If you frequently (and purposefully) leak the data in one-time global initializers, the "minimal" mode is useful for you. Otherwise, should avoid it for stricter modes.

" Normal "Heap-checking tracks   live Objects   and reports a leak For any data which is not reachable via a live object when the program exits.

" Strict "heap-checking are much like" normal "but have a few extra checks that memory isn ' t lost in Globa L destructors. In particular, if you had a global variable that allocates memory during program execution, and then "forgets" about the Memory in the global destructor (say, by setting the pointer-to-it-to-NULL) without freeing it, that would prompt a leak me Ssage in "strict" mode, though "normal" mode.

" draconian "heap-checking is appropriate for those the WHO-to-be very precise about their memory manag Ement, and want the heap-checker to help them enforce it. In "draconian" mode, the Heap-checker does does "live object" checking at all, so it reports a leak unless   all   allocated memory was freed before program exit. (However, you can use   ignoreobject ()   to re-enable liveness-checking in an Object-by-object basis.)

"Normal" mode, as the name implies, is the one used most often at Google. It ' s appropriate for everyday heap-checking use.

In addition, there is and other possible modes:

• as-is
• local

as-is is the most flexible mode; It allows specify the various knobs of the heap checker explicitly. local activates the explicit Heap-check instrumentation, but does no turn on any whole-progra M leak checking.

But very unfortunate. It's still not checked out:

The above disclosure statistics are not expected, due to "leaking" at least 10G of memory.

Then it is mandatory to release the unused buffer:

Mallocextension::instance ()->releasefreememory ();

The problem has been overcome.

Linux Debugging (ix) "Memory leak" in a production environment