Kernel debugging systemtap

Source: Internet
Author: User
Tags gpg systemtap
    • 1. First check your kernel version

Related Technologies: utrace, probe, ftrace, dtrace, pstrace

    • Install systemtap on centos
    • Use instance
    • Reference

Original Text connection

Reference connection

The key point is kernel matching.

1. First check your kernel version


2.6.18-194. EL5


If your kernel version is old, you need to find the kernel-devel version you need.

If you want to use yum for installation, you can use Yum install kernel-devel


Note: The latest kernel-devel version is installed in the latest yum. The kernel-devel version must match the kernel version.

For example, centos release 5.5 (final) uses kernel version 2.6.18-194. EL5.

First download the corresponding

Then install



After kernel-devel is installed, the kernel directory is located in

/Usr/src/kernels/2.6.18-194. el5-i686/


2. Install systemtap

Yum install systemtap (STAP-V 1.3/0.137 please install elfutils-devel to prompt which version of debuginfo you need to install)


3. Install debuginfo


Find and youKernel exact match

(For example, the local uname-r 2.6.18-194. EL5 can only be selected.







If you cannot find the RPM of the corresponding kernel version, you can search for it by Google. You can use the debuginfo RPM package of RedHat.


RedHat kernel reference download


Ftp:// OS /x86_64/Debuginfo/kernel-debuginfo-2.6.18-128.el5.x86_64.rpm

Install rpm-IVH kernel-debuginfo *. rpm


How to test:

STAP-ve 'probe begin {log ("Hello World") Exit ()}'



  Officially recommended installation commands
2 Yum install systemtap kernel-devel Yum-utils
Debuginfo-install KernelBecause no kernel-debuginfo package exists on my centos Yum repository, You have to manually add the repository with this package. Of course, you can also download this package and its dependent package RPM file to your local device for installation.
In/Etc/yum. Repos. D to create a new file, run the command debuginfo
Then add it to debuginfo.
6 [debuginfo]
Name = centos-$ releasever-debuginfo
Base url =
Gpgcheck = 0
Enabled = 1
Gpgkey = file: // etc/pki/rpm-GPG/RPM-GPG-KEY-CentOS-5 and then execute the officially recommended installation command. It takes a long time to download over 300 m files.


Official wiki:



How does systemtap track libc. So?

In the afternoon, I had a hard timeProgramMemory leakage problem. valgrind, gogle perftools and other tools are not very easy to use, so it is easy to get rid of the application, so we plan to use systemtap to understand the memory usage at the libc. So layer. The main idea is to check the balance between the number of malloc/realloc and free calls.

First, prepare the environment. The system is the standard RHEL 5u4:


$ Uname - R  2.6 . 18 - 164  . EL5 $ STAP - Vsystemtap Translator /Driver (Version 1.3 / 0.137 Non- GIT sources) Copyright (c)  2005 - 2010  Red Hat, Inc. And othersthis is Free Software; see the source For  Copying conditions. enabled features: librpm libsqlite3 NSS boost_shared_ptr tr1_unordered_map $ STAP -L '  Kernel. function ("printk ")  '  Kernel.  Function ( "  Printk @ kernel/printk. C: 533  " ) $ FMT : Char Const *$ ARGs: va_list $ STAP -L '  Process ("/lib64/libc. so.6"). function ("malloc ")  '  Missing separate debuginfos, use: debuginfo - Install Glibc- 2.5 - 42 . X86_64



The kernel symbol is OK, and glibc does not have the installation symbol. The system prompts to use the debuginfo-install glibc-2.5-42.x86_64 command to install the symbolic information, but RHEL 5 does not pay for this service, you can only download the package installation.


$ Wget -C FTP Slave- 2.5 - 42  . X86_64.rpm $  Sudo Rpm-I glibc-debuginfo- 2.5 - 42  . X86_64.rpm $ STAP -L '  Process ("/lib64/libc. so.6"). function ("malloc ")  '  Process (  "  /Lib64/ " ). Function ( "  _ Libc_malloc @/usr/src/debug/glibc-2.5-20061008T1257/malloc. C: 3560  " ) $ Bytes: size_t



With the glibc symbol, you can easily track the usage of malloc in libc. So.
Next we will simply write a C program to call malloc, and write a STAP script to track the call stack of malloc:

$ Cat  T. C # include <Stdlib. h> Void fun () {malloc (  1000  );}  Int Main (Int Argc, Char * Argv []) {fun (); Return  0  ;} $  Cat  M. STP probe process (  "  /Lib64/libc. so.6  " ). Function ( "  Malloc  "  ){  If (Target () = PID () {print_ubacktrace (); exit () ;}} probe begin {println (  "  ~  "  );} $  Gcc - G t. C $ STAP -L '  Process ("./A. Out"). function ("*")  '  Process (  "  /Home/chuba/a. Out  " ). Function ( "  Fun @/home/chuba/T. C: 3  "  ) Process (  "  /Home/chuba/a. Out  " ). Function ( "  Main @/home/chuba/T. C: 7  " ) $ Argc: Int $ Argv: Char **



Now that the program is ready, let's execute it to see where the memory leaks:

$SudoStap m. STP-C ./A. Out~0x33d5e74b96: Malloc +0x16/Zero X 230[Libc-2.5. So]0x4004a6[A. out +0x4a6/Zero X 1000]


We can see that malloc is called in 0x4004a6 of A. Out, but what is the specific line in the program? It is easy to find out with add2line:

$ Addr2line-E./A. Out 0x4004a6 /Home/chuba/T. C: 5  $  NL  T. C 1 # Include <stdlib. h> 2  Void fun (){  3 Malloc ( 1000  );  4  }  5    Int Main ( Int Argc, Char * Argv []) {  6  Fun (); 7 Return 0  ;  8 }



Reference connection

Erlang non-amateur research

IBM Linux self-check and systemtap

A new performance measurement and tuning diagnostic tool systemtap series in Linux

Systemtap Manual

RedHat systemtap introduce (PDF) Library Link

Ftp:// OS /i386/Debuginfo

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.