Remember once Tcmalloc allocated memory Coredump

Phenomenon

The service on the line appears coredump, and the stack is:

#0 0x000000000045d145 in Getstacktrace (void**, int, int.) () #1 0x000000000045ec22 in Tcmalloc::P ageheap::growheap (unsign Ed long) () #2 0x000000000045eeb3 in Tcmalloc::P ageheap::new (unsigned long) () #3 0x0000000000459ee8 in Tcmalloc::centralf  Reelist::P opulate () () #4 0x000000000045a088 in Tcmalloc::centralfreelist::fetchfromspanssafe () () #5 0x000000000045a10a in Tcmalloc::centralfreelist::removerange (void**, void**, Int.) () #6 0x000000000045c282 in Tcmalloc: : Threadcache::fetchfromcentralcache (unsigned long, unsigned long) () #7 0x0000000000470766 in Tc_malloc () #8 0x00007f755  32cd4c2 in __conhash_get_rbnode (node=0x22c86870, hash=30) at build/release64/cm_sub/conhash/conhash_inter.c:88#9 0x00007f75532cd76e in __conhash_add_replicas (CONHASH=0X24FBC7E0, Iden=<value optimized out>) at Build/releas E64/cm_sub/conhash/conhash_inter.c:45#10 0X00007F75532CD1FA in Conhash_add_node (CONHASH=0X24FBC7E0, iden=0) at build /release64/cm_sub/conhash/conhash.c:72#11 0x00007f75532c651b in Cm_sub::topocluster::initlbpolicyinfo (this=0x2593a400) at build/release64/cm_sub/topo_cluster.cpp:114#12 0x00007f75532cad73 in Cm_sub::topoclustermanager::p rocessclustermaptable (THIS=0XA219E0, ref=0x267ea8c0) at build/ Release64/cm_sub/topo_cluster_manager.cpp:396#13 0x00007f75532c5a93 in Cm_sub::subrespmsgprocess::reinitcluster ( THIS=0X9C2F00, msg=0x4e738ed0) at build/release64/cm_sub/sub_resp_msg_process.cpp:157 ...

The data structure of the application layer is reviewed, and there is no problem with the basic information. So the initial suspicion is that tcmalloc internal maintenance of the wrong memory, in allocating memory error, this stack is only the appearance of the problem. A few days later, another service on the line, based on the same library, also the core, the stack is the same.

The initial positioning problem starts with something recently updated, including a dependent server environment, but there is no obvious problem, so you can only start with the direct cause of the core.

Analysis Getstacktrace

Confirm the detailed location of the core:

# Core in this directive 0x000000000045d145 <_z13getstacktraceppvii+21>: mov    0x8 (%rax),%r9 (gdb) p/x $rip              # core Instruction location $ 9 = 0x45d145 (gdb) p/x $rax              $ = 0x4e73aa58 (gdb) x/1a $rax +0x8         # Rax + 8 = 0x4e73aa600x4e73aa60:     0x0

The instruction attempts to read the contents from [0x4e73aa60] and then error, the memory unit is unreadable. But what exactly does this instruction mean in the code, which needs to be mapped to the code . Get the source code for Tcmalloc and find that GetStackTrace there are many implementations based on the compilation options, so choose the most likely implementation here, and then compare the assembly to confirm that the code matches. The first choice was stacktrace_x86-64-inl.h to find a complete mismatch and then choose stacktrace_x86-inl.h . Support for 64-bit platforms is also available in this implementation version.

stacktrace_x86-inl.hSome macros are used to generate the function names and parameters, and the code is roughly the following:

int Get_stack_trace_or_frames {      void **sp;      unsigned long RBP;      __asm__ volatile ("mov%%rbp,%0": "=r" (RBP));      SP = (void * *) RBP;      int n = 0;      while (SP && n < max_depth) {        if (* (sp+1) = = Reinterpret_cast<void *> (0)) {break          ;        }        void **next_sp = Nextstackframe<!is_stack_frames, is_with_context> (sp, UCP);        if (Skip_count > 0) {          skip_count--;        } else {          result[n] = * (sp+1);          n++;        }        SP = next_sp;      }      return n;    }

NextStackFrameis a template function that contains a lot of code, which is very simple after simplification:

Template<bool strict_unwinding, bool with_context>    static void **nextstackframe (void **old_sp, const void *UC ) {      void **new_sp = (void * *) *old_sp;      if (strict_unwinding) {        if (new_sp <= old_sp) return NULL;        if ((uintptr_t) new_sp-(uintptr_t) old_sp > 100000) return NULL;      } else {        if (new_sp = = old_sp) return NULL;        if ((new_sp > Old_sp)            && ((uintptr_t) new_sp-(uintptr_t) old_sp > 1000000)) return NULL;      }      if ((uintptr_t) new_sp & (sizeof (void *)-1)) return NULL;      return new_sp;    }

The above code to the compilation of the comparison process still took some time, in which some constants appearing in the assembly can greatly shorten the contrast time, for example, the above appears 100000 , the assembly is:

0x000000000045d176 <_z13getstacktraceppvii+70>: cmp    $0x186a0,%rbx  # 100000=0x186a0

Note that the NextStackFrame if (STRICT_UNWINDING) template parameter is used, which causes the generated code to have no else part at all, and there is no 1000000 such constant

In the process of comparing the code, you can know the key registers, the memory location corresponding to the variables in the code, so that the core can be restored in the field environment . The analysis process does not have to be read from the first line assembly, can be read from a more obvious location, thereby restoring the entire code, function return instructions, jump instructions, comparison instructions, read memory instructions, parameters register , etc. are relatively obvious corresponding places.

Also note that GetStackTrace in the RecordGrowth call, 3 parameters are passed in:

Getstacktrace (T->stack, kMaxStackDepth-1, 3); Kmaxstackdepth = 31

Here is a simple note of my analysis:

(GDB) Disassembledump of assembler code for function _z13getstacktraceppvii:0x000000000045d130 <_ Z13getstacktraceppvii+0&gt: Push%rbp0x000000000045d131 <_z13getstacktraceppvii+1>: mov%rsp,%rbp0x000000000 045d134 <_z13getstacktraceppvii+4&gt: Push%rbx0x000000000045d135 <_z13getstacktraceppvii+5>: mov%rbp,%r ax0x000000000045d138 <_z13getstacktraceppvii+8>: Xor%r8d,%r8d0x000000000045d13b <_Z13GetStackTracePPvii+ 11&GT: Test%rax,%rax0x000000000045d13e <_z13getstacktraceppvii+14>: je 0x45d167 <_z13getstacktraceppvii+5 5>0x000000000045d140 <_z13getstacktraceppvii+16>: CMP%esi,%r8d # while (.. max_depth > n? 0x000000 000045d143 <_z13getstacktraceppvii+19>: Jge 0x45d167 <_z13getstacktraceppvii+55>0x000000000045d145 < _Z13GETSTACKTRACEPPVII+21&GT: mov 0x8 (%rax),%R9 # key location: * (sp+1), R9, rax corresponding SP variable 0x000000000045d149 <_z13gets Tacktraceppvii+25>: Test%R9,%R9 # * (sp+1) = = 0 0x000000000045d14c <_z13getstacktraceppvii+28>: je 0x45d167 <_z13getstacktraceppvii+55>0x0000000 00045D14E <_z13getstacktraceppvii+30>: mov (%rax),%RCX # new_sp = *old_sp, this is already nextstackframe code 0x0000000000 45d151 <_z13getstacktraceppvii+33>: CMP%rcx,%rax # new_sp <= old_sp? 0x000000000045d154 <_z13getstacktraceppvii+36&gt: JB 0x45d170 <_Z13GetStackTracePPvii+64> # new_sp > Old _SP Jump 0x000000000045d156 <_z13getstacktraceppvii+38>: Xor%ecx,%ecx0x000000000045d158 <_ Z13getstacktraceppvii+40&gt: Test%edx,%edx # skip_count > 0? 0x000000000045d15a <_z13getstacktraceppvii+4 2&gt: Jle 0x45d186 <_z13getstacktraceppvii+86>0x000000000045d15c <_z13getstacktraceppvii+44>: Sub $0x1,  %edx # skip_count--0x000000000045d15f <_z13getstacktraceppvii+47>: mov%rcx,%rax 0x000000000045d162 <_z13getstacktraceppvii+50&gt: Test%rax,%rax # while (sp? 0x000000000045d165 <_z13getstacktraceppvii+53>: jne 0x45d140 <_z13getstacktraceppvii+16>0x000000000045d167 <_z13getstacktraceppvii+55&gt: Pop%rbx0x000000000045d168 <_z13getstacktraceppvii+56>: Leaveq 0x000000000045d169 <_z13getstacktraceppvii+57>: mov%r8d,%eax # r8 stores the return value, r8=n0x000000000045d16c <_Z13G Etstacktraceppvii+60&gt: RETQ # return n0x000000000045d16d <_z13getstacktraceppvii+61>: Nopl ( %rax) 0x000000000045d170 <_z13getstacktraceppvii+64>: mov%rcx,%rbx 0x000000000045d173 <_Z13GetStackTrac     Eppvii+67&gt: Sub%rax,%rbx # offset = new_sp-old_sp0x000000000045d176 <_z13getstacktraceppvii+70>: CMP $0X186A0,%RBX # offset > 100000 0x000000000045d17d <_z13getstacktraceppvii+77>: ja 0x45d156 <_Z13G Etstacktraceppvii+38> # return null0x000000000045d17f <_z13getstacktraceppvii+79>: Test $0X7,%CL # New_ SP & (sizeof (void*)-1) 0x000000000045d182 <_z13getstacktraceppvii+82>: je 0x45d158 <_z13getstacktraceppvii+40>0x000000000045d184 <_ Z13getstacktraceppvii+84&gt: jmp 0x45d156 <_z13getstacktraceppvii+38>0x000000000045d186 <_    Z13getstacktraceppvii+86&gt: MOVSLQ%r8d,%rax # rax = n0x000000000045d189 <_z13getstacktraceppvii+89>: add $0x1,%r8d # n++0x000000000045d18d <_z13getstacktraceppvii+93>: mov%r9, (%rdi,%rax,8) # key location: result[n] = * (sp+1) 0x000000000045d191 <_z13getstacktraceppvii+97>: jmp 0x45d15f <_Z13GetStackTracePPvii+47>

The analysis process is time-consuming and can also analyze the GetStackTrace implementation principle of the next function, in fact, the use of the RBP register constantly backtracking, so as to get the entire call stack functions of the address (strictly speaking the return address). Simply indicate the RBP in the function call:

   ... saved registers          # i.e push Rbxlocal variabes           # i.e Sub 0x10, Rspreturn address # call           xxxlast func RBP            # push RBP; mov rsp, rbpsaved registerslocal variables return addresslast func RBP ...                      # RSP

In short, in general, in any one function, the RBP register points to the stack base address of the current function, which in turn stores the caller's stack base, and also stores the return address of the caller in front of the stack base. So, GetStackTrace the implementation, in simple terms, is probably:

SP = RBP  //Gets the stack base of the current function Getstacktrace while    (n < max_depth) {        new_sp = *sp        Result[n] = * (new_sp+1) 
   n++    }

Above, you will finally know the following key information:

• The R8 corresponds to the variable n, which indicates the current fetch to the first stack frame.
• Rax corresponding variable SP, code core in * (SP+1)
• RDI corresponding variable result, used to store each address obtained

Then you can see what the scene is like:

(gdb) x/10a $rdi 0x1ffc9b98:     0x45a088 <_ZN8tcmalloc15CentralFreeList18FetchFromSpansSafeEv+40>       0x45a10a <_zn8tcmalloc15centralfreelist11removerangeeppvs2_i+106>0x1ffc9ba8:     0x45c282 <_ Zn8tcmalloc11threadcache21fetchfromcentralcacheemm+114>      0x470766 <TC_MALLOC+790>0X1FFC9BB8:     0X7F75532CD4C2 <__conhash_get_rbnode+34>        0x00x1ffc9bc8:     0x0     0x00x1ffc9bd8:     0x0     0x0 ( GDB) p/x $r 8$3 = 0x5 (gdb) p/x $rax $4 = 0x4e73aa58

Summary:

GetStackTraceAn error occurred while fetching the called __conhash_get_rbnode function, and 5 function addresses were obtained. The RBP currently in use is 0x4e73aa58 .

Wrong RBP.

RBP is also taken out of the stack, since this address has a problem, the first thought is that there is code local variable/array write out of bounds. For example sprintf , use. Also, the general write-over-break stack may be destroying the caller's stack , for example:

Char s[32];memcpy (S, p, 1024);

Because writes are written from a low address to a high address, and the caller's stack is at a high address. Of course, you will also encounter the stack of callers who write bad callers, that is, cross-stack frames out of bounds, such as previously encountered:

Len = vsnprintf (buf, sizeof (BUF), FMT, wtf-long-string); Buf[len] = 0;

__conhash_get_rbnodeThe RBP is taken from the Tcmalloc stack:

(GDB) F 7#7  0x0000000000470766 in Tc_malloc () (GDB) x/10a $rsp 0x4e738b80:     0x4e73aa58      0x22c868700x4e738b90 :     0x4e738bd0      0x850x4e738ba0:     0x4e73aa58      0x7f75532cd4c2 <__conhash_get_rbnode+34>   # 0x4e73aa58

So there's the suspicion that tcmalloc there is a stack break in this function, and this is the time to read the code and see if there are any suspected dangers. This is where the deadlock, the suspicion of cross-stack frame destruction, can only be called in the __conhash_get_rbnode stack around the function, such as __conhash_get_rbnode the function called the __conhash_add_replicas string operation is exactly:

void __conhash_add_replicas (conhash_t *conhash, int32_t iden)    {        node_t* node = __conhash_create_node (Iden, Conhash->replica);        ...        Char Buf[buf_len]; Buf_len = ...        snprintf (buf, Buf_len, virt_node_hash_fmt, Node->iden, i);        uint32_t hash = Conhash->cb_hashfunc (BUF);        if (Util_rbtree_search (& (Conhash->vnode_tree), hash) = = NULL)        {            util_rbtree_node_t* rbnode = __ Conhash_get_rbnode (node, hash);            ...

This code finally found that there is no problem, and it took a lot of time. Later found in a number of functions of the RBP is a bit strange, this call stack is more normal range is: 0X4E738C90

(GDB) F 8#8  0x00007f75532cd4c2 in __conhash_get_rbnode (node=0x22c86870, hash=30) (gdb) p/x $rbp $6 = 0x4e73aa58     # That's not particularly suspicious. (GDB) F 9#9  0x00007f75532cd76e in __conhash_add_replicas (CONHASH=0X24FBC7E0, Iden=<value optimized out>) (gdb) p/x $rbp $7 = 0x4e738c60     # This is not particularly suspicious (GDB) F 10#10 0x00007f75532cd1fa in Conhash_add_node (CONHASH=0X24FB C7E0, iden=0) at build/release64/cm_sub/conhash/conhash.c:72 (gdb) p/x $rbp      # suspicious $8 = 0x0 (gdb) F 11#11 0x00007f75532c65 1b in Cm_sub::topocluster::initlbpolicyinfo (this=0x2593a400) (gdb) p/x $rbp      # suspicious $9 = 0X2598FEF0

Why do RBP in many functions look abnormal? If you think about it, how clever is it if the stack is broken in the code?

Source of error RBP

Then the transfer came, and suddenly flashed in my mind -fomit-frame-pointer . The compiler generates code that does not require a stack base address pointer, that is, the RBP register is not a stack base address register. Most functions, or functions that frame-pointer are open, have the following directives for their function heads:

Push   %rbpmov    %rsp,%rbp ...

Represents saving the caller's stack base address to the stack, and setting its own stack base address. Look at the __conhash series of functions;

Dump of assembler code for function __conhash_get_rbnode:0x00007f75532cd4a0 <__conhash_get_rbnode+0>:    mov    %rbx,-0x18 (%RSP) 0x00007f75532cd4a5 <__conhash_get_rbnode+5>:    mov    %rbp,-0x10 (%RSP) ...

This library is compiled separately, no display specified -fno-omit-frame-pointer , consult the GCC Manual, O2 optimization is turned on omit-frame-pinter .

In the absence of RBP, Tcmalloc's GetStackTrace attempt to read RBP fetch the return address of the call, is naturally problematic. However, if the function in the entire call stack is either RBP or no RBP, GetStackTrace the result of the removal is to skip some stack frames, without error. unless one of these functions uses the RBP register for another purpose (the compiler saves the register, it must be used separately). So here we continue to trace the 0x4e73aa58 source of the wrong address.

The source is already quite obvious, it must be __conhash_get_rbnode set in, because the RBP of this function is saved in the callee tcmalloc .

Dump of assembler code for function __conhash_get_rbnode:0x00007f75532cd4a0 <__conhash_get_rbnode+0>:    mov    %rbx,-0x18 (%RSP) 0x00007f75532cd4a5 <__conhash_get_rbnode+5>:    mov    %rbp,-0x10 (%RSP) 0X00007F75532CD4AA <__conhash_get_rbnode+10>:   mov    %esi,%ebp                    # rewritten RBP0X00007F75532CD4AC <__ Conhash_get_rbnode+12>:   mov    %r12,-0x8 (%RSP) 0x00007f75532cd4b1 <__conhash_get_rbnode+17>:   Sub    $0x18,%rsp0x00007f75532cd4b5 <__conhash_get_rbnode+21>:   mov    %rdi,% R120x00007f75532cd4b8 <__conhash_get_rbnode+24>:   mov    $0x30,%edi0x00007f75532cd4bd <__conhash_ Get_rbnode+29>:   callq  0x7f75532b98c8 <[email protected]>  # call Tcmalloc, assemble here

The value printed here for the RSI register may be misleading, because any time the value of the print register may be wrong, unless it has been shown to be saved. But here you can see that the value of the RSI comes from the parameter (RSI corresponds to the second parameter):

void __conhash_add_replicas (conhash_t *conhash, int32_t iden)    {        node_t* node = __conhash_create_node (Iden, Conhash->replica);        ...        Char Buf[buf_len]; Buf_len = ...        snprintf (buf, Buf_len, virt_node_hash_fmt, Node->iden, i);        uint32_t hash = Conhash->cb_hashfunc (BUF); The hash value is computed by a string hash function        if (Util_rbtree_search (& (Conhash->vnode_tree), hash) = = NULL)        {            Util_rbtree _node_t* Rbnode = __conhash_get_rbnode (node, hash);  Hash value ...            

Catch up __conhash_add_replicas :

0x00007f75532cd764 <__conhash_add_replicas+164>:        mov    %ebx,%esi    # from rbx0x00007f75532cd766 <_ _conhash_add_replicas+166>:        mov    %r15,%rdi0x00007f75532cd769 <__conhash_add_replicas+169>:        CALLQ  0x7f75532b9e48 <[email protected]> (gdb) p/x $rbx $11 = 0x4e73aa58 (gdb) p/x hash$12 = 0x4e73aa58      # 0x4e 73aa58

Found 0x4e73aa58 the source. This address value is actually a string hash algorithm calculated! Here you can also look at the contents of this string:

(gdb) x/1s $rsp 0x4e738bd0:      "conhash-00000-00133"

The hash function of this bunker is conhash_hash_def .

Conditions of Coredump

Above, since as long as a certain library omit-frame-pointer , that tcmalloc can be wrong, why the frequency is not high? This can be returned to GetStackTrace NextStackFrame the implementation in particular, which contains several legal RBP judgments:

if (new_sp <= old_sp) return NULL;  The RBP of the previous stack frame is certainly greater than the current large        if ((uintptr_t) new_sp-(uintptr_t) old_sp > 100000) return NULL;//pointer value range must also be within 100000 ...        C3/>if ((uintptr_t) new_sp & (sizeof (void *)-1)) return NULL; Since the pointer itself is saved, it must also be an integer multiple of sizeof (void*), aligned

With these conditions, the core probability becomes very low.

Summarize

Finally, if you are familiar with Tcmalloc, the whole problem is estimated to be second: Tcmalloc INSTALL

Report

Another interesting thing to attach.

At the __conhash_add_replicas time of parsing, a 64-byte array of characters is defined to view its stack:

(gdb) x/20a $rsp 0x4e738bd0:     0x2d687361686e6f63      0x30302d3030303030          # These are the strings CONHASH-00000-001330X4E738BE0:     0x333331        0x00x4e738bf0:     0x0     0x7f75532cd69e <__conhash_ CREATE_NODE+78>0X4E738C00:     0x24fbc7e0      0x4e738c600x4e738c10:     0x24fbc7e0      0x7f75532cd6e3 <_ _CONHASH_ADD_REPLICAS+35>0X4E738C20:     0x0     0x24fbc7e80x4e738c30:     0x4e738c20      0X24FBC7E00X4E738C40:     0x22324360      0X246632C00X4E738C50:     0x0     0x00x4e738c60:     0x0     0X7F75532CD1FA <conhash_add_node+74>

At first I thought it was buf 64 bytes, the whole [0x4e738bd0, 0X4E738C10] memory, but there was a function address in this memory, which once made me suspect there was a problem. Later wake up these addresses are defined buf before __conhash_create_node the call generated, the call process is written to the stack, after the call stack pointer changes, but do not need to empty the contents of the stack.

Original address: http://codemacro.com/2015/04/06/tcmalloc-getstacktrace/
Written by Kevin Lynx posted athttp://codemacro.com

Remember once Tcmalloc allocated memory Coredump