Linux Debugging (ix) "Memory leak" in a production environment

An accidental opportunity to discover that a process uses more than 14G of memory. This process is an RPC server, just as a transit, absolutely should not use so much memory. Even if there is too much concurrency and there is too much data in memory, this memory usage will definitely come down in the case of concurrency reduction. But in fact, this memory will go up until it is killed by Oom killer.

Because this RPC server logic is relatively simple, the first daytime source code, in addition to find some simple programming above the problem, there is no big problem. First on Valgrind:

Valgrind--tool=memcheck--leak-check=full-v./rpc_server

The original case is that the memory leaks are generally checked out. Not this time (otherwise there will be no article):

In fact at least 10G of "memory leaks". Since it's not checked out, it means that the memory is still alive. Imagine this scenario: each request is a new piece of memory, placed in a list. After the normal request is processed, the memory needs to be removed from this list. If not deleted, then this is a memory leak. But Valgrind can't check it out.

Because the above process uses the Tcmalloc, is not tcmalloc problem? We know that tcmalloc efficiency is better than malloc, then it is not tcmalloc problem, if it has been applied for memory, does not release, it will cause this "memory leak." Note the following passage:

Releasing memory back to the Systemby default, Tcmalloc would release no-longer-used Memory back to the kernel gradually, O Ver time. The TCMALLOC_RELEASE_RATE flag controls how quickly this happens. You can also force a release at a given point in the progam execution like so:   mallocextension::instance ()->release Freememory (); can also call Setmemoryreleaserate () to change the Tcmalloc_release_rate value at runtime, or getmemoryreleaserate to See what's the current release rate is.

A simple translation is a mechanism for tcmalloc memory back into the OS: By default, Tcmalloc will return unused memory for a long time to the system. Tcmalloc_release_rate This flag controls the frequency of this turn-back. You can force this release to occur at run time by this statement:

Mallocextension::instance ()->releasefreememory ();

Of course, you can pass. SetMemoryReleaseRate() To set this tcmalloc_release_rate. If set to 0, the representation is never delivered back. The larger the number represents the greater the frequency of the return. The general reasonable value is to set a number between 0-10. You can also set environment variables by setting thetcmalloc_release_rateTo set this rate.

With this suspicion, first check the heap usage using Google's gpreftools:

1. Export Heapcheck=draconian

2. Export Pprof_path=/usr/local/bin/pprof

You can start directly.

is set to draconian , because you want more detailed statistics. More will believe the explanation is as follows: Flavors ofHeap Checking

these is the legal values when running a whole-program heap check:

1. minimal
2. normal
3. strict
4. Draconian

"Minimal" heap-checking starts as late as possible in a initialization, meaning can leak some memory in your Initiali Zation routines (that run before   main () , say), and not Trigger a leak message. If you frequently (and purposefully) leak the data in one-time global initializers, the "minimal" mode is useful for you. Otherwise, should avoid it for stricter modes.

"Normal" Heap-checking tracks   live Objects   and reports a leak for any data so is no T reachable via a live object when the program exits.

"Strict" heap-checking are much like "normal" but have a few extra checks that memory isn ' t lost in global destructors. In particular, if you had a global variable that allocates memory during program execution, and then "forgets" about the Memory in the global destructor (say, by setting the pointer-to-it-to-NULL) without freeing it, that would prompt a leak me Ssage in "strict" mode, though "normal" mode.

"draconian" heap-checking is appropriate-those who-like-be very precise about their memory management, and want th e Heap-checker to help them enforce it. In "draconian" mode, the Heap-checker does does "live object" checking at all, so it reports a leak unless   all   allocated memory was freed before program exit. (However, you can use   ignoreobject ()   to re-enable liveness-checking in an Object-by-object basis.)

"Normal" mode, as the name implies, is the one used most often at Google. It ' s appropriate for everyday heap-checking use.

In addition, there is and other possible modes:

• as-is
• local

as-is is the most flexible mode; It allows specify the various knobs of the heap checker explicitly. local activates the explicit Heap-check instrumentation, but does no turn on any whole-progra M leak checking.

Unfortunately, it's still not checked out:

The above disclosure statistics are not expected because "leaks" at least 10G of memory.

Then it is mandatory to release the unused buffer:

Mallocextension::instance ()->releasefreememory ();

The problem is solved.

Linux Debugging (ix) "Memory leak" in a production environment