Linux program crash debugging technology

Author: Song Wensheng
 
I. Cause
 
During android ril development, rild crashes abnormally. This process directly controls all operations related to android RIL. if an exception is terminated, the android framework restarts.
 
Ii. Details
 
A) As we all know, when a linux program crashes, the stack trace before the crash is printed. This stack trace is an important clue for us to find the cause of the crash.
 
B) The following are the crash details of android rild.
 
01-19 17:48:56. 550 I/DEBUG (683): Build fingerprint: 'augusta/M70P/m70p/miracle_smt: 2.2/M70P-daily/20120119: user/test-keys'
01-19 17:48:56. 550 I/DEBUG (683): pid: 684, tid: 715 >>>/ system/bin/rild <
01-19 17:48:56. 550 I/DEBUG (683): signal 11 (SIGSEGV), fault addr deadbaad
01-19 17:48:56. 550 I/DEBUG (683): r0 00000000 r1 0000000c r2 00000027 r3 00000000
01-19 17:48:56. 550 I/DEBUG (683): r4 afd40328 r5 deadbaad r6 00001728 r7 00000000
01-19 17:48:56. 550 I/DEBUG (683): r8 00100000 r9 ae60532d 10 10000000 fp 00000000
01-19 17:48:56. 550 I/DEBUG (683): ip ffffffff sp 100ffcf0 lr afd154e1 pc afd11dee cpsr 00000030
01-19 17:48:56. 560 W/InputManagerService (887): Window already focused, ignoring focus gain of: com. android. internal. view. IInputMethodClient $ Stub $ Proxy @ 45ffbd88
01-19 17:48:56. 580 I/DEBUG (683): #00 pc 00011dee/system/lib/libc. so
01-19 17:48:56. 580 I/DEBUG (683): #01 pc worker be1c/system/lib/libc. so
01-19 17:48:56. 580 I/DEBUG (683 ):
01-19 17:48:56. 580 I/DEBUG (683): code around pc:
01-19 17:48:56. 580 I/DEBUG (683): afd11dcc 2d00d10d 1c2bd00b 2d00682d e026d1fb
01-19 17:48:56. 580 I/DEBUG (683): afd11ddc 2b0068db 4e17d003 51a02001 4d164798
01-19 17:48:56. 580 I/DEBUG (683): afd11dec 702a2227 edfef7fb f7fc2106 2380ef1c
01-19 17:48:56. 580 I/DEBUG (683): afd11dfc aa010559 60912400 1c116054 94012006
01-19 17:48:56. 580 I/DEBUG (683): afd11e0c eaa0f7fc 2200a905 f7fc2002 f7fbeaac
01-19 17:48:56. 580 I/DEBUG (683 ):
01-19 17:48:56. 580 I/DEBUG (683): code around lr:
01-19 17:48:56. 580 I/DEBUG (683): afd154c0 447b4a0d 589cb083 26009001 686768a5
01-19 17:48:56. 580 I/DEBUG (683): afd154d0 220ce008 2b005eab 1c28d003 47889901
01-19 17:48:56. 590 I/DEBUG (683): afd154e0 35544306 d5f43f01 2c006824 b003d1ee
01-19 17:48:56. 590 I/DEBUG (683): afd154f0 bdf01c30 0002ae62 000000d4 1c0fb5f0
01-19 17:48:56. 590 I/DEBUG (683): afd15500 43551c3d a904b087 1c16ac01 604d9004
01-19 17:48:56. 590 I/DEBUG (683 ):
01-19 17:48:56. 590 I/DEBUG (683): stack:
01-19 17:48:56. 590 I/DEBUG (683): 100ffcb0 00000718
01-19 17:48:56. 590 I/DEBUG (683): 100ffcb4 afd1455b/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffcb8 afd405a0/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100 ffcbc afd4054c/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffcc0 00000000
01-19 17:48:56. 590 I/DEBUG (683): 100ffcc4 afd154e1/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffcc8 00000071
01-19 17:48:56. 590 I/DEBUG (683): 100 ffccc afd1452d/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffcd0 00000000
01-19 17:48:56. 590 I/DEBUG (683): 100ffcd4 afd40328/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffcd8 00000000
01-19 17:48:56. 590 I/DEBUG (683): 100 ffcdc 00001728
01-19 17:48:56. 590 I/DEBUG (683): 100ffce0 00000000
01-19 17:48:56. 590 I/DEBUG (683): 100ffce4 afd147cb/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffce8 df002777
01-19 17:48:56. 590 I/DEBUG (683): 100 ffcec e3a070ad
01-19 17:48:56. 590 I/DEBUG (683): #00 100ffcf0 listen b4e8 [heap]
01-19 17:48:56. 590 I/DEBUG (683): 100ffcf4 c0000000
01-19 17:48:56. 590 I/DEBUG (683): 100ffcf8 afd418dc/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100 ffcfc afd10538/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffd00 afd40328/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffd04 fffffbdf
01-19 17:48:56. 590 I/DEBUG (683): 100ffd08 afd40328/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffd0c afd11224/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffd10 20.b000 [heap]
01-19 17:48:56. 590 I/DEBUG (683): 100ffd14 afd0be21/system/lib/libc. so
01-19 17:48:56. 600 I/DEBUG (683): #01 100ffd18 afd40328/system/lib/libc. so
01-19 17:48:56. 600 I/DEBUG (683): 100ffd1c afd0be21/system/lib/libc. so
01-19 17:48:56. 600 I/DEBUG (683): 100ffd20 00000002
01-19 17:48:56. 600 I/DEBUG (683): 100ffd24 running b4ee [heap]
01-19 17:48:56. 600 I/DEBUG (683): 100ffd28 release limit 4 [heap]
01-19 17:48:56. 600 I/DEBUG (683): 100ffd2c rjb4e8 [heap]
01-19 17:48:56. 600 I/DEBUG (683): 100ffd30 release limit 4 [heap]
01-19 17:48:56. 600 I/DEBUG (683): 100ffd34 00000006
01-19 17:48:56. 600 I/DEBUG (683): 100ffd38 10813fc
01-19 17:48:56. 600 I/DEBUG (683): 100ffd3c af9059ff/system/lib/libcutils. so
01-19 17:48:56. 600 I/DEBUG (683): 100ffd40 492b4e8 [heap]
01-19 17:48:56. 600 I/DEBUG (683): 100ffd44 800056b5/system/lib/libaugusta-ril.so
01-19 17:48:56. 600 I/DEBUG (683): 100ffd48 100ffd6c
01-19 17:48:56. 600 I/DEBUG (683): 100ffd4c rjb438 [heap]
01-19 17:48:56. 600 I/DEBUG (683): 100ffd50 ae6089bc/system/lib/libril. so
01-19 17:48:56. 600 I/DEBUG (683): 100ffd54 afd0cd81/system/lib/libc. so
01-19 17:48:56. 600 I/DEBUG (683): 100ffd58 800056b5/system/lib/libaugusta-ril.so
01-19 17:48:56. 600 I/DEBUG (683): 100ffd5c ae6042a3/system/lib/libril. so
 
 
 
 
 
 
 
 
 
 
 
 
 
Iii. Analysis
 
Fortunately, we can know the cause of the crash in the first three lines of this log. But more details need to be checked by ourselves.
 
Stack record:
 

 
01-19 17:48:56. 590 I/DEBUG (683): stack:
01-19 17:48:56. 590 I/DEBUG (683): 100ffcb0 00000718
01-19 17:48:56. 590 I/DEBUG (683): 100ffcb4 afd1455b/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffcb8 afd405a0/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100 ffcbc afd4054c/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffcc0 00000000
01-19 17:48:56. 590 I/DEBUG (683): 100ffcc4 afd154e1/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffcc8 00000071
01-19 17:48:56. 590 I/DEBUG (683): 100 ffccc afd1452d/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffcd0 00000000
01-19 17:48:56. 590 I/DEBUG (683): 100ffcd4 afd40328/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffcd8 00000000
01-19 17:48:56. 590 I/DEBUG (683): 100 ffcdc 00001728
01-19 17:48:56. 590 I/DEBUG (683): 100ffce0 00000000
01-19 17:48:56. 590 I/DEBUG (683): 100ffce4 afd147cb/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffce8 df002777
01-19 17:48:56. 590 I/DEBUG (683): 100 ffcec e3a070ad
01-19 17:48:56. 590 I/DEBUG (683): #00 100ffcf0 listen b4e8 [heap]
01-19 17:48:56. 590 I/DEBUG (683): 100ffcf4 c0000000
01-19 17:48:56. 590 I/DEBUG (683): 100ffcf8 afd418dc/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100 ffcfc afd10538/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffd00 afd40328/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffd04 fffffbdf
01-19 17:48:56. 590 I/DEBUG (683): 100ffd08 afd40328/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffd0c afd11224/system/lib/libc. so
01-19 17:48:56. 590 I/DEBUG (683): 100ffd10 20.b000 [heap]
01-19 17:48:56. 590 I/DEBUG (683): 100ffd14 afd0be21/system/lib/libc. so
01-19 17:48:56. 600 I/DEBUG (683): #01 100ffd18 afd40328/system/lib/libc. so
01-19 17:48:56. 600 I/DEBUG (683): 100ffd1c afd0be21/system/lib/libc. so
01-19 17:48:56. 600 I/DEBUG (683): 100ffd20 00000002
01-19 17:48:56. 600 I/DEBUG (683): 100ffd24 running b4ee [heap]
01-19 17:48:56. 600 I/DEBUG (683): 100ffd28 release limit 4 [heap]
01-19 17:48:56. 600 I/DEBUG (683): 100ffd2c rjb4e8 [heap]
01-19 17:48:56. 600 I/DEBUG (683): 100ffd30 release limit 4 [heap]
01-19 17:48:56. 600 I/DEBUG (683): 100ffd34 00000006
01-19 17:48:56. 600 I/DEBUG (683): 100ffd38 10813fc
01-19 17:48:56. 600 I/DEBUG (683): 100ffd3c af9059ff/system/lib/libcutils. so
01-19 17:48:56. 600 I/DEBUG (683): 100ffd40 492b4e8 [heap]
01-19 17:48:56. 600 I/DEBUG (683): 100ffd44 800056b5/system/lib/libaugusta-ril.so
01-19 17:48:56. 600 I/DEBUG (683): 100ffd48 100ffd6c
01-19 17:48:56. 600 I/DEBUG (683): 100ffd4c rjb438 [heap]
01-19 17:48:56. 600 I/DEBUG (683): 100ffd50 ae6089bc/system/lib/libril. so
01-19 17:48:56. 600 I/DEBUG (683): 100ffd54 afd0cd81/system/lib/libc. so
01-19 17:48:56. 600 I/DEBUG (683): 100ffd58 800056b5/system/lib/libaugusta-ril.so
01-19 17:48:56. 600 I/DEBUG (683): 100ffd5c ae6042a3/system/lib/libril. so
 
Register:
 
01-19 17:48:56. 550 I/DEBUG (683): r0 00000000 r1 0000000c r2 00000027 r3 00000000
01-19 17:48:56. 550 I/DEBUG (683): r4 afd40328 r5 deadbaad r6 00001728 r7 00000000
01-19 17:48:56. 550 I/DEBUG (683): r8 00100000 r9 ae60532d 10 10000000 fp 00000000
01-19 17:48:56. 550 I/DEBUG (683): ip ffffffff sp 100ffcf0 lr afd154e1 pc afd11dee cpsr 00000030
 
Offset of the PC in the module:
 
01-19 17:48:56. 580 I/DEBUG (683): #00 pc 00011dee/system/lib/libc. so
01-19 17:48:56. 580 I/DEBUG (683): #01 pc worker be1c/system/lib/libc. so
 
Machine code near PC:
 
01-19 17:48:56. 580 I/DEBUG (683): code around pc:
01-19 17:48:56. 580 I/DEBUG (683): afd11dcc 2d00d10d 1c2bd00b 2d00682d e026d1fb
01-19 17:48:56. 580 I/DEBUG (683): afd11ddc 2b0068db 4e17d003 51a02001 4d164798
01-19 17:48:56. 580 I/DEBUG (683): afd11dec 702a2227 edfef7fb f7fc2106 2380ef1c
01-19 17:48:56. 580 I/DEBUG (683): afd11dfc aa010559 60912400 1c116054 94012006
01-19 17:48:56. 580 I/DEBUG (683): afd11e0c eaa0f7fc 2200a905 f7fc2002 f7fbeaac
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Machine code near LR:
 
01-19 17:48:56. 580 I/DEBUG (683): code around lr:
01-19 17:48:56. 580 I/DEBUG (683): afd154c0 447b4a0d 589cb083 26009001 686768a5
01-19 17:48:56. 580 I/DEBUG (683): afd154d0 220ce008 2b005eab 1c28d003 47889901
01-19 17:48:56. 590 I/DEBUG (683): afd154e0 35544306 d5f43f01 2c006824 b003d1ee
01-19 17:48:56. 590 I/DEBUG (683): afd154f0 bdf01c30 0002ae62 000000d4 1c0fb5f0
01-19 17:48:56. 590 I/DEBUG (683): afd15500 43551c3d a904b087 1c16ac01 604d9004
 
The preceding parts are composed of logs. The role of each part is explained below.
 
Stack record:
 
1. Role of stacks in programs
 
A) pass function parameters
 
B) record the return address of the Function
 
C) temporary data storage
 
2. From the stack record, we can see the call process between functions.
 
3. From the above log, we can see that libril calls the function in libaugusta-ril, and then calls the function of other modules.
 
Register:
 
1. Registers are the basic unit of computer operations. People familiar with assembly languages are no stranger to registers. Some registers are General registers, while others are specialized registers with their own specific functions. The specific register content is not described here.
 
2. PC is an important register in registers. Maintains the position of the Code being executed in the memory space of the currently running program. In this example, the content of the pc is afd11dee, indicating that when the program runs afd11dee, a problem occurs.
 
Note that the value shown here is the actual program memory space value, not the offset in the module.
 
3. LR is a register similar to a PC. It retains the value of the PC when the program jumps. People who have read the Linux kernel and started to guide the assembler should understand the role of lr.
 
Machine code: the machine code is not very useful for personal purposes and can be used to assist in positioning.
 
4. Hands-on
 
After understanding the functions of each part of the log, we need to find the real cause of the crash.
 
When a Linux program is running, it loads all the modules used to the memory, and all the segments are distributed to the unified virtual memory space. From the above stack log, we can see the call process of the program, where the address is the virtual address of the memory space. We only know the module in which the location is located, but we do not know which function has a problem. However, the address actually corresponds to a function. Only by knowing the module's memory distribution can the function at the corresponding offset be found.
 
1. View Memory Distribution
 
Our process name is rild. Run the ps-ef command to obtain the pid of the process, which is assumed to be 702. Under/proc, the running state information of each process is displayed. Run cat/proc/702/maps to obtain the memory distribution for the process.
 
20178000-0000a000 r-xp 00000000 00: 0e 718/system/bin/rild
 
2017a000-0000b000 rw-p 00002000 00: 0e 718/system/bin/rild
 
2017b000-0000e000 rw-p 00000000 0 [heap]
 
80000000-8000b000 r-xp 00000000 00: 0e 311/system/lib/libaugusta-ril.so
 
8000b000-8000c000 rw-p rjb000 00: 0e 311/system/lib/libaugusta-ril.so
 
A7e00000-a7e04000 r-xp 00000000 00: 0e 240/system/lib/libhardware_legacy.so
 
A7e04000-a7e05000 rw-p 00004000 00: 0e 240/system/lib/libhardware_legacy.so
 
A8100000-a8127000 r-xp 00000000 00: 0e 236/system/lib/libutils. so
 
A8127000-a8128000 rw-p 00027000 00: 0e 236/system/lib/libutils. so
 
A8200000-a821f000 r-xp 00000000 00: 0e 269/system/lib/libbinder. so
 
A821f000-a8225000 rw-p 0001f000 00: 0e 269/system/lib/libbinder. so
 
Ae300000-ae304000 r-xp 00000000 00: 0e 297/system/lib/libnetutils. so
 
Ae304000-ae305000 rw-p 00004000 00: 0e 297/system/lib/libnetutils. so
 
Ae400000-ae402000 r-xp 00000000 00: 0e 233/system/lib/libwpa_client.so
 
Ae402000-ae403000 rw-p 00002000 00: 0e 233/system/lib/libwpa_client.so
 
Ae600000-ae608000 r-xp 00000000 00: 0e 349/system/lib/libril. so
 
Ae608000-ae609000 rw-p 00008000 00: 0e 349/system/lib/libril. so
 
Af700000-af714000 r-xp 00000000 00: 0e 326/system/lib/libz. so
 
Af714000-af715000 rw-p 00014000 00: 0e 326/system/lib/libz. so
 
Af900000-af90e000 r-xp 00000000 00: 0e 264/system/lib/libcutils. so
 
Af90e000-af90f000 rw-p rje000 00: 0e 264/system/lib/libcutils. so
 
Afa00000-afa03000 r-xp 00000000 00: 0e 261/system/lib/liblog. so
 
Afa03000-afa04000 rw-p 00003000 00: 0e 261/system/lib/liblog. so
 
Afb00000-afb20000 r-xp 00000000 00: 0e 356/system/lib/libm. so
 
Afb20000-afb21000 rw-p 00020000 00: 0e 356/system/lib/libm. so
 
Afc00000-afc01000 r-xp 00000000 00: 0e 323/system/lib/libstdc ++. so
 
Afc01000-afc02000 rw-p 00001000 00: 0e 323/system/lib/libstdc ++. so
 
Afd00000-afd40000 r-xp 00000000 00: 0e 332/system/lib/libc. so
 
Afd40000-afd43000 rw-p 00040000 00: 0e 332/system/lib/libc. so
 
B0001000-b0011000 r-xp 00001000 00: 0e 735/system/bin/linker
 
B0011000-b0012000 rw-p 00011000 00: 0e 735/system/bin/linker
 
Bee51000-bee53000 rw-p 00000000 0 [stack]
 
2. Location
 
Note that the above memory information is obtained from the stack trace log on two machines, so there is a deviation.
 
The above maps shows the distribution of each module in the virtual space.
 
With the module distribution, we can locate the functions in the module.
 
From the stack log, found 800056b5/system/lib/libaugusta-ril.so, from maps found that the starting position of the libaugusta-ril.so is 80000000, it is easy to know that this function is located at 00056b5 of the module.
 
Now, we have learned how to locate the function location in the module. The next step is to obtain the function name.
 
5. Search
 
When the system compiles a program, it is divided into several steps. The final step is strip, which removes unnecessary symbols and debugging information from the generated module to reduce the volume and loading time. Therefore, if we use the final so directly, we will not get the desired result. What we need is a module without strip.
 
During android compilation, the generated intermediate files are stored in the obj directory of the out directory. In the obj directory, each module has its own folder. In the module folder, there is a LINKED Directory, which keeps the result of no strip. This file is exactly what we need.
 
Vi. Dump and Decompilation
 
Any compilation tool provides a dump tool to understand the internal information of the generated results.
 
With dump, we can know the segment information, symbol table information, and even disassembly of the generated program.
 
Here we use arm-eabi-objdump-dx libaugusta-ril.so> dumplog for disassembly.
 
 
 
Search for 56b5 from the dumplog file, and we find that the function corresponding to this location is onRequest.
 
According to the above method, we will analyze it step by step to know the function call path in case of a problem and clearly understand the cause of the problem.
 
VII. Inspiration of this example
 
The above log information is caused by a special operation and is also prone to errors. Here is a detailed description.
 
The final error was found on the free function of libc. so. Based on this clue, the code was re-viewed based on the function call path and the problem was found.
 
The error code model is as follows:
 
Struct a {char * B ;};
 
 
 
Void func2 (struct a * s)
 
{S. B + = 1 ;}
 
 
 
Void func1 ()
 
{
 
Struct a st;
 
St. B = malloc (100 );
 
Func2 (& st );
 
Free (st. B );
 
}
 
 
 
The address of st is passed to func2 as the pointer parameter, but member B is modified and takes effect in func2. The final result is that when func1 is free of the pointer, an exception occurs because the pointer position is different During allocation and release.
 
When designing code, we should try to avoid similar use. If it is unavoidable, we should make comments.