Debugging techniques for native Process Memory leakage in Android

Debugging techniques for native Process Memory leakage in Android
Honglang blog

The Code is based on android2.3.x.

Android provides a convenient memory leak information and tools (such as mat) for Java programs to facilitate searching. However, for natvie processes written purely in C/C ++, it is not easy to find memory leaks. Traditional C/C ++ programs can use the valgrind tool or some code check tools. Fortunately, Google's bionic Library provides a great API -- get_malloc_leak_info for us to find memory leaks. Using this method, we can easily find the suspected Memory leakage through backtrace.

CodePrincipleAnalysis

We can use ADB shell setprop libc. Debug. malloc 1 to set the debug level of the memory (debug_level). For more details, see the notes in the bionic/libc/bionic/malloc_debug_common.c file:

/* Handle to shared library where actual memory allocation is implemented.
* This library is loaded and memory allocation CILS are redirected there
* When libc. Debug. malloc environment variable contains value other
* Zero:
* 1-for Memory Leak detections.
* 5-for filling allocated/freed memory with patterns defined
* Chk_sentinel_value, and chk_fill_free macros.
* 10-for adding pre-, and post-allocation stubs in order to detect
* Buffer overruns.
* Note that emulator's memory allocation instrumentation is not controlled
* Libc. Debug. malloc value, but rather by emulator, started with-memcheck
* Option. note also, that if emulator has started with-memcheck option,
* Emulator's instrumented memory allocation will take over value saved in
* Libc. Debug. malloc. In other words, if emulator has started with-memcheck
* Option, libc. Debug. malloc value is ignored.
* Actual functionality for debug levels 1-10 is implemented in
* Libc_malloc_debug_leak.so, while functionality for emultor's instrumented
* Allocations is implemented in libc_malloc_debug_qemu.so and can be run inside
* The emulator only.
*/

For different debugging levels, the memory allocation management function operation handle points to different memory allocation management functions. In this way, different function versions are used for memory allocation and release at different debugging levels.
The detailed process is as follows:

As mentioned in the following code comment, in the _ libc_init routine, malloc_debug_init will be called for initialization and malloc_init_impl will be called (in a process, pthread_once is used to ensure that it is executed only once)

In malloc_init_impl, the corresponding C library is opened, and the function symbol malloc_debug_initialize (see row 366) is parsed and executed (row 373)

When debug_level is set to 1, 5, or 10, open the Database "/system/lib/libc_malloc_debug_leak.so ". In the file bionic/libc/bionic/malloc_debug_leak.c, malloc_debug_initialize is implemented, but only an empty function with 0 returned is returned. If the value is 20, "/system/lib/libc_malloc_debug_qemu.so" is enabled"

Next, different memory operation functions malloc/free/calloc/realloc/memalign are parsed for different debug_levels:

For debug_level levels 1, 5, and 10, the implementations of various versions of malloc/free/calloc/realloc/memalign are located in the bionic/libc/bionic/malloc_debug_leak.c file. For example, if debug_level is set to 5, malloc/free/fills the allocated memory with 0xeb and 0xef when releasing the memory:

When debug_level is 1 to debug memory leak, the implementation is to play backtrace:

Void * leak_malloc (size_t bytes)
{
// Allocate enough space infront of the allocation to store the pointer
// The alloc structure. This will making free 'ing the structer really fast!

// 1. allocate enough memory and include our Header
// 2. Set the base pointer to be right after our Header

Void * base = dlmalloc (Bytes + sizeof (allocationentry ));
If (base! = NULL ){
Pthread_mutex_lock (& gallocationsmutex );

Intptr_t backtrace [backtrace_size];
Size_t numentries = get_backtrace (backtrace, backtrace_size );

Allocationentry * Header = (allocationentry *) base;
Header-> entry = record_backtrace (backtrace, numentries, bytes );
Header-> Guard = guard;

// Now increment base to point to after our header.
// This shoshould just work since our header is 8 bytes.
Base = (allocationentry *) base + 1;

Pthread_mutex_unlock (& gallocationsmutex );
}

Return base;
}

This malloc function allocates an additional piece of data before the actually allocated bytes for allocationentry. After the memory is allocated successfully, an array of 32 elements is allocated to store the call stack pointer. The call function get_backtrace saves the call stack, that is, the function pointer is saved to the array backtrace. Then, the call stack is recorded using record_backtrace, And the entry member of allocationentry points to it. The record_backtrace function searches for the hash value in the ghashtable of the global call stack table. If not, create a call stack and add it to the global table. Finally, move back the place pointed to by the base, and then it is the allocated memory address.
It can be seen that the malloc function of this version records the call stack information. By adding a head before the allocated memory block, the entry on How to query the stack call information of the hash table is saved.

Let's take a look at the record_backtrace function. Before analyzing its code, let's take a look at the struct (File malloc_debug_common.h ):
Struct hashentry {
Size_t slot; // slots array index in hashtable
Hashentry * Prev; // the first item.
Hashentry * Next; // The next item. It is added to the end when it is added.
Size_t numentries; // Number of function pointers in the call stack
// Fields above "size" are not sent to the host
Size_t size; // indicates the number of memories allocated for this malloc operation.
Size_t allocations; // number of calls, that is, the number of times the malloc is called
Intptr_t backtrace [0]; // call the stack
};

Typedef struct hashtable;
Struct hashtable {
Size_t count;
Hashentry * slots [hashtable_size]; // hashtable_size = 1543
};
And in a process, there is a global variable ghashtable, which is used to record the list of call stacks that ultimately call the memory allocated by malloc. The type of ghashtable is hashtable, which has a pointer pointing to a slots array with a maximum capacity of 1543. The number of valid values in the array is recorded by count of another member. You can obtain the hash value through backtrace and numentries, and then obtain the index of the hashentry in the array after division with hashtable_size. In this way, you can quickly obtain the index in the Array Based on the hash information.
Another struct is hashentry. Because its members point to the front and back pointers, it is also a linked list. If the hash value is the same, it will be added to the back of the linked list. The first hashentry member slot is its own index in the array, that is, it is calculated by hash. The last one is to call the stack backtrace [0], which is a function pointer, the number of items in this array is recorded by another member numentries; size indicates the size of the allocated memory; Allocations indicates the number of times of allocation, that is, the number of times of the same call path.
The two data structure relationships can be expressed:

When calling record_backtrace in leak_malloc to record stack information, the backtrace and numentries first obtain the hash value, and then get the array index in the ghashtable after the division operation. Then, check whether this item already exists, that is, whether hashentry with the same memory size, the same call path, and a considerable number of function pointers are allocated. If yes, add 1 directly to allocations in the original item. If no, create a new project: allocate memory for the hashentry struct (see row 151, note that the last member backtrace needs to determine the number of items based on the numentries value), and then call the stack information to copy to the last member backtrace of hashentry. Finally, add a count for the entire table.
In this way, the record_backtrace function completes the task of adding backtrace information to the global table: either adding a hashentry or adding an index.

Static hashentry * record_backtrace (intptr_t * backtrace, size_t numentries, size_t size)
{
Size_t hash = get_hash (backtrace, numentries); // obtain the hash values of Backtrace and numentries.
Size_t slot = hash % hashtable_size; // division, the hashentry array index in hashtable is obtained.

If (size & size_flag_mask ){
Debug_log ("malloc_debug: Allocation % ZX exceeds bit widthn", size );
Abort ();
}

If (gmallocleakzygotechild)
Size | = size_flag_zygote_child;

Hashentry * entry = find_entry (& ghashtable, slot, backtrace, numentries, size );
// The previous row: Search for the existence of this item in the global table, that is, whether the call path has been called
If (entry! = NULL ){
Entry-> allocations ++; // if it has been called, the count is increased.
} Else {// if no call is made, a new item is created.
// Create a new entry
Entry = (hashentry *) dlmalloc (sizeof (hashentry) + numentries * sizeof (intptr_t); // allocate memory for this item,
If (! Entry) // connect to a row: because the last entry of hashentry is intptr_t backtrace [0], it is a dynamic length, all numentries * sizeof (intptr_t)
Return NULL;
Entry-> allocations = 1;
Entry-> slot = slot;
Entry-> Prev = NULL;
Entry-> next = ghashtable. Slots [Slot];
Entry-> numentries = numentries;
Entry-> size = size;

Memcpy (Entry-> backtrace, backtrace, numentries * sizeof (intptr_t); // copy the backtrace to the memory behind the entry struct

Ghashtable. Slots [Slot] = entry; // Add a new assigned hashentry to the array in hashtable.

If (Entry-> next! = NULL ){
Entry-> next-> Prev = entry;
}

// We just added an entry, increase the size of the hashtable
Ghashtable. Count ++; // increase the count
}

Return entry;
}

The stack items in the above global hash table are released in the leak_free function (see row 550 ):

Void leak_free (void * MEm)
{
If (MEM! = NULL ){
Pthread_mutex_lock (& gallocationsmutex );

// Check the guard to make sure it is valid
Allocationentry * Header = (allocationentry *) mem-1;

If (header-> guard! = Guard ){
// Cocould be a memaligned Block
If (void **) MEm) [-1] = memalign_guard ){
Mem = (void **) MEm) [-2];
Header = (allocationentry *) mem-1;
}
}

If (header-> Guard = guard | is_valid_entry (header-> entry )){
// Decrement the allocations
Hashentry * entry = header-> entry;
Entry-> allocations -;
If (Entry-> allocations <= 0 ){
Remove_entry (entry );
Dlfree (entry );
}

// Now free the memory!
Dlfree (header );
} Else {
Debug_log ("Warning bad header guard: '0x % x '! And Invalid entry: % pn ",
Header-> guard, header-> entry );
}

Pthread_mutex_unlock (& gallocationsmutex );
}
}

Therefore, the remaining unreleased items in the global table are the call stacks that call malloc when memory is allocated but not released.

Get_malloc_leak_info

The get_malloc_leak_info function is used to obtain Memory leakage information. When allocating memory, record the call stacks and clear them when releasing them. In this way, the rest may be the root cause of Memory leakage. So how can I obtain the global hash table of the stack called by the memory? The get_malloc_leak_info function is provided in the malloc_debug_common.c file to obtain the stack information.
The get_malloc_leak_info function receives five parameters for various addresses that store various variables. After the call, these variables are modified. As the code comment says:
* Info will point to the entire memory allocated in the function. The memory size is overallsize;
The entire space is composed of several small items. The size of each item is infosize. The data structure of this small item is equivalent to the structure starting with the size member in the hashentry, that is, the first member is the memory size allocated by malloc, the second member is allocations, that is, the count with the same call Stack multiple times, and the last one is backtrace, which contains 32 (backtrace_size) pointer values. Therefore, the large memory block pointed to by * info contains a total of small items such as overallsize and infosize. Note that the backtrace array in hashentry is allocated according to the actual quantity. In this case, the space is allocated by 32 in a unified manner. If less than 32, the value following is set to 0;
Totalmemory is the size of all memory allocated by malloc;
The last parameter is backtracesize, that is, 32 (backtrace_size)

The get_malloc_leak_info function first checks whether the passed variables are legal and whether stack items exist in the global Stack:
Void get_malloc_leak_info (uint8_t ** info, size_t * overallsize,
Size_t * infosize, size_t * totalmemory, size_t * backtracesize)
{
// Don't do anything if we have invalid arguments
If (Info = NULL | overallsize = NULL | infosize = NULL |
Totalmemory = NULL | backtracesize = NULL ){
Return;
}
* Totalmemory = 0;

Pthread_mutex_lock (& gallocationsmutex );

If (ghashtable. Count = 0 ){
* Info = NULL;
* Overallsize = 0;
* Infosize = 0;
* Backtracesize = 0;
Goto done;
}

Check the number of items in the global stack table, and allocate a piece of memory to save the pointer. These pointers are used to point to all hashentry items in the ghashtable, by the way, the total number of allocated but unreleased memories is counted. totalmemory is used to return to the caller. The last parameter is the number of function pointers in the call stack. The actual value is backtrace_size, that is, 32 ..
Void ** list = (void **) dlmalloc (sizeof (void *) * ghashtable. Count );

// Get the entries into an array to be sorted
Int Index = 0;
Int I;
For (I = 0; I Hashentry * entry = ghashtable. Slots [I];
While (entry! = NULL) {// put the valid items in the list
List [Index] = entry;
* Totalmemory = * totalmemory + // calculate the total allocated memory
(Entry-> size &~ Size_flag_mask) * entry-> allocations );
Index ++;
Entry = entry-> next; // point the entry to the next, that is, the same slot value.
}
} // After this for loop, all the stack item pointers in the global table are stored in the table pointed to by list.

// XXX: The Protocol doesn't allow variable size for the stack trace (yet)
* Infosize = (sizeof (size_t) * 2) + (sizeof (intptr_t) * backtrace_size); // 32 pointer values,
// Note: There are two size_t variables in front of info. They are the size and allocations members in hashentry, followed by backtrace.
* Overallsize = * infosize * ghashtable. Count; // calculate the memory required for all call stack items
* Backtracesize = backtrace_size;

Finally, allocate memory for all call stack item information, that is, the place where info points to, and call stack information in ghashtable (that is, the hashentry in the List table is the value after the size of its struct member) copy to the memory pointed to by info.

// Now get a byte array big enough for this
* Info = (uint8_t *) dlmalloc (* overallsize); // allocate memory for all stack items, including two size_t variables of each item

If (* info = NULL) {// allocation failed, no memory
* Overallsize = 0;
Goto out_nomem_info;
}

Qsort (void *) List, ghashtable. Count, sizeof (void *), hash_entry_compare); // sort the items in the list

Uint8_t * head = * Info;
Const int COUNT = ghashtable. count;
For (I = 0; I <count; I ++ ){
Hashentry * entry = list [I];
Size_t entrysize = (sizeof (size_t) * 2) + (sizeof (intptr_t) * entry-> numentries );
If (entrysize <* infosize ){
/* We're writing less than a full entry, clear out the rest */
Memset (Head + entrysize, 0, * infosize-entrysize); // unfilled part of the call Stack's 32 Items
} Else {
/* Make sure the amount we're copying doesn' t exceed the limit */
Entrysize = * infosize;
} // The following row copies the space occupied by 32 pointers plus the values of the first two size_t variables to the info item.
Memcpy (Head, & (Entry-> size), entrysize); // The size_t variables are size and allocations, respectively.
Head + = * infosize; // point the head to the memory of the next info
}

Out_nomem_info:
Dlfree (list );

Done:
Pthread_mutex_unlock (& gallocationsmutex );
}

When the program ends, the memory should be released. In this case, we can call get_malloc_leak_info to obtain the unreleased call stack item. In principle, these are the places where memory leaks. However, the actual situation may be that some memory should be retained and should not be released when we run get_malloc_leak_info.
In addition, we sometimes want to check that the process is a daemon and will not exit. Therefore, some memory should be kept and not released. At this time, we can select a time point in a certain state to view the unreleased memory. For example, when the idle status is just entered, we can use get_malloc_leak_info to obtain unreleased memory information, then, when the program returns the idle status after some operations are completed, use get_malloc_leak_info again to obtain unreleased memory information. Comparing the two types of information, the new call stack items are suspected of Memory leakage.
The sample code for using the get_malloc_leak_info function is as follows:

Typedef struct {
Size_t size; // allocated memory
Size_t dups; // Number of duplicates
Intptr_t * backtrace; // call the stack pointer
} Allocentry;

Uint8_t * info = NULL;
Size_t overallsize = 0;
Size_t infosize = 0;
Size_t totalmemory = 0;
Size_t backtracesize = 0;

Get_malloc_leak_info (& info, & overallsize, & infosize, & totalmemory, & backtracesize );
Logi ("returned from get_malloc_leak_info, info = 0x % x, overallsize = % d, infosize = % d, totalmemory = % d, backtracesize = % d", (INT) info, overallsize, infosize, totalmemory, backtracesize );
If (Info ){
Uint8_t * PTR = Info;
Size_t COUNT = overallsize/infosize;

Snprintf (buffer, size, "allocation count % in", count );
Result. append (buffer );
Snprintf (buffer, size, "total meory % in", totalmemory );
Result. append (buffer );

Allocentry * entries = new allocentry [count]; // Array

For (size_t I = 0; I <count; I ++) {fill the obtained stack information in the allocentry array.
// Each entry shocould be size_t, size_t, intptr_t [backtracesize]
Allocentry * E = & entries [I];

E-> size = * reinterpret_cast <size_t *> (PTR );
PTR + = sizeof (size_t );

E-> dups = * reinterpret_cast <size_t *> (PTR );
PTR + = sizeof (size_t );

E-> backtrace = reinterpret_cast <intptr_t *> (PTR );
PTR + = sizeof (intptr_t) * backtracesize;
}

Specific debugging steps:
Reference http://freepine.blogspot.com/2010/02/analyze-memory-leak-of-android-native.html
Download its patch package and Python Toolkit
Patch the code to frameworks/base in the android source code, re-compile and generate an image, and burn it into the mobile phone board. Then there will be a binary program memorydumper under/system/bin. The code patch package adds a service to the mediaserver process, and the binary program uses the service through binder IPC. This service uses get_malloc_leak_info to obtain the unreleased memory information.

Step1. set the debugging level and restart the mediaserver process.
ADB shell setprop libc. Debug. malloc 1
ADB shell PS mediaserver
ADB Shell kill <mediaserver_pid>

It aims to allow the mediaserver process to use the leak_malloc version. When the debugging level is set, the mediaserver process is killed, and the Android system restarts it automatically. In this case, it reloads the libc library, and the memory allocation function uses the leak_malloc and leak_free versions through handle.
Step 2: Execute memorydumper in an initial state, such as before using the camera program, and record the memory not released at this time:
$ ADB shell/system/bin/memorydumper
$ ADB pull/data/memstatus _ <mediaserver_pid>. 0.

Step 3: perform some operations, such as taking a photo, recording a video, or playing several songs, and then exit these applications;

Step 4: Execute memorydumper again to record the memory that has not been released at this time. Compare the differences between memorydumper and Step 2 using comparison tools. These differences are suspected of Memory leakage. Because the first possible reason for unrelease is that it should not be released at that time, and the comparison is to exclude them.
$ ADB pull/data/memstatus _ <mediaserver_pid>. 1.
$ Diff memstatus _ <mediaserver_pid>. 0 memstatus _ <mediaserver_pid>. 1> diff_0_1

Step 5: Obtain the maps file. Based on this file, we can get the address range space of the. So library file, which is used to parse the symbolic address of the Call Stack function.
$ ADB pull/proc/<mediaserver_pid>/maps your_path

Step 5: Execute the Python script in the reference link:
./Addr2func. py-root-Dir = ~ /U8500-android-2.3_v4.30-maps-file = maps-Product = u8500 diff. _ 0_1> memleak. backtrace
This script analyzes the maps file to obtain the address space occupied by the library file corresponding to the address segment, and obtains the library corresponding to the address of each call stack. Run the following command, obtain the corresponding function name, source file, and row number after the mangled Compiler:
[Root-Dir]/prebuilt/linux-x86/toolchain/arm-eabi-4.4.0/bin/arm-eabi-addr2line-f-e [root-Dir] // out/target/product/[product]/symbols/ [libname] callstack_address

Then demangle the function using [root-Dir]/prebuilt/linux-x86/toolchain/arm-eabi-4.4.0/bin/ARM-Eabi-C ++ filt to get the function name consistent with the source code, this makes it easier to recognize.

Example of snapshot:
The following is the call stack address obtained by using memorydumper for the first time:

The following is the call stack address obtained by using memorydumper for the second time:

The differences between the two are obtained after diff comparison:

Call Stack obtained after addr2func is used:

Link: http://www.redwolf-blog.com /? P = 1233

Original article, copyright honglang blog all, reproduced at will, but please indicate the source.