Linux addr2line Application Guide

Debugging Linux applications on zSeries is similar to debugging Linux applications on other architectures. The biggest challenge for experienced Linux developers is to understand the new system architecture. It seems a daunting task for mainframe developers who are new to Linux to master new debugging tools. Don't be afraid. This article provides some useful tips for Linux addr2line to help you get started.

UserDebug Logging

The first step in debugging a crashed program is to find out what went wrong. The Linux Kernel on zSeries has such a built-in feature that records some basic debugging information when a user's process crashes. To enable this feature, run the following command as the root user:
Echo1>/proc/sys/kernel/userprocess_debug

When a process crashes, additional information will be provided in the log file/var/log/messages, including the cause of program termination, fault address, and the program status word PSW) brief register dump for General registers and access registers.
 

 
 

  
  
Mar3111:34:28l02kernel:Userprocessfault:interruptioncode0x10  
  
  
Mar3111:34:28l02kernel:failingaddress:0  
  
  
Mar3111:34:28l02kernel:CPU:1  
  
  
Mar3111:34:28l02kernel:Processsimple(pid:30122,stackpage=05889000)  
  
  
Mar3111:34:28l02kernel:  
  
  
Mar3111:34:28l02kernel:UserPSW:070dc000c00ab738  
  
  
Mar3111:34:28l02kernel:task:05888000ksp:05889f08pt_regs:05889f68  
  
  
Mar3111:34:28l02kernel:UserGPRS:Mar3111:34:28l02kernel:00000000004019a0004019a000000000  
  
  
Mar3111:34:28l02kernel:00000003c00ab732004008f800400338  
  
  
Mar3111:34:28l02kernel:40018ffc0040061c40018e347ffff800  
  
  
Mar3111:34:28l02kernel:00400434804006248040066e7ffff800  
  
  
Mar3111:34:28l02kernel:UserACRS:  
  
  
Mar3111:34:28l02kernel:00000000000000000000000000000000  
  
  
Mar3111:34:28l02kernel:00000001000000000000000000000000  
  
  
Mar3111:34:28l02kernel:00000000000000000000000000000000  
  
  
Mar3111:34:28l02kernel:00000000000000000000000000000000  
  
  
Mar3111:34:28l02kernel:UserCode:  
  
  
Mar3111:34:28l02kernel:4440500007fea74a0001185418431835a8240000  
 
 

The above indicates that the program name is "simple") terminating the operating system with a program interrupt code 0x10 indicates that this is a segment Conversion error), and the fault address is 0. There is no doubt that a null pointer is used. Now that we know what happened, we need to find out where it happened.

Basic diagnosis of Linux addr2line

The information provided by the UserDebug log entries can be used to determine the program crash location. Some available tools can help solve various program termination problems you may encounter. We will gradually introduce those tools in this article.

First, let's check the user PSW in the log entry. The PSW contains the command address, Status Code, and other information about the machine status. Currently, we only care about the instruction address between 33rd and 63rd bits ). For simplicity, let's assume that the user PSW is 070dc00080400618. Remember, we are investigating an ESA/39031-bit addressing) PSW. 32nd bits are not part of the instruction address. They indicate the 31-bit addressing mode, but must be processed when studying the PSW value. To obtain the actual instruction pointer, subtract the second word of PSW from 0x80000000. The result is a command address 0x400618. To locate the code, you need some information in the executable file. First, use readelf to print some program header information.
 

 
 

  
  
ElffiletypeisEXEC(Executablefile)Entrypoint0x400474Thereare6programheaders,startingatoffset52ProgramHeaders:  
  
  
TypeOffsetVirtAddrPhysAddrFileSizMemSizFlgAlign  
  
  
PHDR0x0000340x004000340x004000340x000c00x000c0RE0x4  
  
  
INTERP0x0000f40x004000f40x004000f40x0000d0x0000dR0x1[Requestingprograminterpreter:/lib/ld.so.1]  
  
  
LOAD0x0000000x004000000x004000000x009900x00990RE0x1000  
  
  
LOAD0x0009900x004019900x004019900x000fc0x00114RW0x1000  
  
  
DYNAMIC0x0009ac0x004019ac0x004019ac0x000a00x000a0RW0x4  
  
  
NOTE0x0001040x004001040x004001040x000200x00020R0x4  
  
  
SectiontoSegmentmapping:SegmentSections...  
  
  
00  
  
  
01.interp  
  
  
02.interp.note.ABI-tag.hash.dynsym.dynstr.gnu.version  
  
  
.gnu.version_r.rela.got.rela.plt.init.plt.text.fini.rodata  
  
  
03.data.eh_frame.dynamic.ctors.dtors.got.bss  
  
  
04.dynamic  
  
  
05.note.ABI-tag  
 
 

The preceding results show readelf-lsimple. Remember that "simple" is the name of our test program ). In the ProgramHeaders section, the first LOAD row provides information about where the program is loaded. In the Flg column, this segment is marked as R (read) E (executable ). VirtAddr is the address where the program starts to load. MemSiz is the code length that is being loaded into this segment. Add it to mongoaddr. The basic address range of this program is 0x400000-0x400990. The command address for program crash is 0x400618, which is within the scope of program loading. Now we know that the problem occurs directly in the code.

If the executable file contains debugging symbols, you can determine which line of code causes the problem. Use the addr2line program for the address and executable file as follows:
Addr2line-esimple0x400618

Will return:
/Home/devuser/simple. c: 34

To study this problem, check the 34th rows.

For the original Linux addr2line program crash, PSW is 070dc000c00ab738. To obtain the command address, subtract 0x80000000. The result is 0x400ab738. This address is not exactly within our applet. So what is it? Is the code from the shared library. If you run the ldd command lddsimple on an executable file, the list of shared objects required for running the program and the available address of the library will be returned.
Libc. so.6 =>/lib/libc. so.6 (0x40021000)/lib/ld. so.1 =>/lib/ld. so.1 (0x40000000)

The command address corresponds to the address for loading libc. so.6. In our simple test case, we only need two shared objects. Other applications may need more shared objects, which makes ldd output more complex. We will use perl as an example. Input:
Ldd/usr/bin/perl

You will get:
 

 
 

  
  
libnsl.so.1=> 
  
  
/lib/libnsl.so.1(0x40021000)libdl.so.2=> 
  
  
/lib/libdl.so.2(0x40039000)libm.so.6=> 
  
  
/lib/libm.so.6(0x4003d000)libc.so.6=> 
  
  
/lib/libc.so.6(0x40064000)libcrypt.so.1=> 
  
  
/lib/libcrypt.so.1(0x4018f000)/lib/ld.so.1=> 
  
  
/lib/ld.so.1(0x40000000)  
 
 

Everything is needed, but I find that the following content reads faster for this process:
 

 
 

  
  
ldd/usr/bin/perl|awk‘{print?$4““$3}’  
  
  
|sort(0x40000000)/lib/ld.so.1(0x40021000)  
  
  
/lib/libnsl.so.1(0x40039000)  
  
  
/lib/libdl.so.2(0x4003d000)  
  
  
/lib/libm.so.6(0x40064000)  
  
  
/lib/libc.so.6(0x4018f000)  
  
  
/lib/libcrypt.so.1  
 
 

Now let's determine where the Linux addr2line crash occurs in libc. Assume that the loading address of libc. so.6 is 0x40021000, and the command address 0x400ab738 is subtracted. The result is 0x8a738. This is the offset to enter libc. so.6. Run the nm command to dump the symbols from libc. so.6 and then try to determine the function in which the address is located. For libc. so.6, nm generates more than 7,000 rows of output. By executing the grep regular expression to find the calculated offset, you can reduce the amount of data that must be checked. Input:
Nm/lib/libc. so.6 | sort | grep0008a

66 rows will be returned. In the middle of the output, we will find:
0008a6fcTmemcpy0008a754t_wordcopy_fwd_aligned

This offset is located in a certain position in memcpy. In this example, a null pointer is passed to memcpy as the target address. Where can we call memcpy? Good question. We can determine the target region by checking the register dump output in the log file. Register 14 contains the return address for executing a function call. According to Figure 1, R14 is 0x8040066e, which generates an address 0x40066e after the high position is intercepted. This address falls within the scope of our program, so we can run addr2line to determine where the address is. Input:
Addr2line-esimple0x40066e

Will return:
/Home/devuser/simple. c: 36

This is the line after we call memcpy. One note about addr2line: If the executable file does not contain debugging symbols, you will get ?? : 0 as the response.