Debugging the Linux kernel with Ftrace, part 1th

Ftrace is a debugging tool available in the Linux kernel. Using Ftrace can keep track of what is happening in the kernel, which is useful when debugging bugs or parsing the kernel. This series of articles introduces Ftrace, which is divided into three parts. This article is the first part, introduced the kernel-related compilation options, user-state access to Ftrace interface, ftrace data files, and Ftrace provides a description of the use of the tracker to enable readers to better understand and use the tool.

Ftrace is a tracking tool built into the Linux kernel, starting with 2.6.27 to join the mainstream kernel. Use Ftrace to debug or analyze what is happening in the kernel. Ftrace provides different trackers for use in different situations, such as tracking kernel function calls, tracking context switches, viewing how long interrupts are closed, tracking latency in the kernel state, and performance issues. System developers can use Ftrace to track and debug the kernel to find the root cause of problems in the kernel and to easily fix them. In addition, readers who are interested in the kernel can also observe the activities occurring in the kernel through ftrace, and understand how the kernel works.

Let the kernel support Ftrace

With Ftrace, you first compile it into the kernel. The Kernel/trace/makefile file in the kernel source directory gives the Ftrace related compilation options.

Listing 1. Ftrace list of related configuration options

Config_function_tracer  config_function_graph_tracer  config_context_switch_tracer  CONFIG_NOP_TRACER  config_sched_tracer  ...

Ftrace related configuration options are more, and for different trackers have their own configuration options. Different options have different dependencies, and the Kernel/trace/kconfig file under the kernel source directory describes these dependencies. Readers can refer to the Makefile file and the Konfig file, and then select the tracker they want.

It is often more intuitive to use make menuconfig when configuring the kernel. For example, in the 2.6.33.1 kernel, to compile the ftrace into the kernel, you can select the Tracers menu item under Kernel Hacking (Figure 1) (Figure 2).

Figure 1. Kernel Hacking Figure 2. Tracers

Go to the Tracers menu and see the list of trackers supported by the kernel. As shown in 3, all trackers are selected, and readers can select specific trackers according to their needs.

Figure 3. List of trackers supported by the kernel

Note here that if you are on a 32-bit x86 machine, do not select the Optimize for Size option under the General Setup menu item (Figure 4) at compile time (Figure 5), or you will not see the Kernel Function graph T in Figure 3 Racer option. This is because in the Konfig file, for a 32-bit x86 machine, the table entry Function_graph_tracer has a special dependency condition:

        Depends on! x86_32 | | ! Cc_optimize_for_size

Figure 4. General Setup Figure 5. Optimize for size

The Ftrace provides the access interface to the user state through Debugfs, so the Debugfs is also compiled into the kernel. To activate support for Debugfs, you can edit the kernel Profile. config, set the config_debug_fs=y, or select support for the hacking file system from the Kernel debugfs menu when make menuconfig. As shown in 6.

Figure 6. DEBUGFS compilation options

After the configuration is complete, compile and install the new kernel and boot to the new kernel. Note that when Ftrace support is activated, the compiler's-PG option is used when compiling the kernel, as defined in the Kernel/trace/makefile file, as shown in Listing 2.

Listing 2. Activating compilation Options-PG

Ifdef config_function_tracer  Orig_cflags: = $ (kbuild_cflags)  kbuild_cflags = $ (subst-pg,,$ (ORIG_CFLAGS))  ...  endif ...  

Using the-PG option adds a lot of debugging information to the compiled kernel image. In general, just activate ftrace support during the development testing phase to debug the kernel and fix bugs. The kernel that is eventually used for the release version will turn off the-PG option and will not be able to use Ftrace.

Back to top of page

Access Ftrace via Debugfs

The Ftrace provides the access interface to the user state via the Debugfs. When you configure the kernel, the directory/sys/kernel/debug is created when DEBUGFS is activated, and the Debugfs file system is mounted to that directory. To mount the directory, you need to add the following to the/etc/fstab file:

        Debugfs  /sys/kernel/debug  debugfs  defaults  0  0

Or can be mounted at run time:

        Mount-  t  debugfs  nodev  /sys/kernel/debug

An tracing directory/sys/kernel/debug/tracing is created under DEBUGFS after the support for Ftrace is activated. This directory contains the Ftrace control and output files, shown in 7. Depending on the settings for the ftrace when compiling the kernel, the files and directories actually displayed in this directory are different from here.

Figure 7. Files under the tracing directory

Back to top of page

Ftrace Data files

There are many files and directories in the/sys/kernel/debug/trace directory, some are shared by various trackers, and some are specific to a tracker. When you manipulate these data files, you typically use the echo command to modify their values, or you can manipulate the values of those files in your program by reading and writing related functions in the file. The following is a description of only some of the files, readers can refer to the kernel source package in the Documentation/trace directory and the source files under Kernel/trace to understand the purpose of the remaining files.

• The README file provides a brief description of the operation of the Ftrace command sequence. You can view the file by using the Cat command to understand the operational flow of the profile.
• current_tracer is used to set or display the tracker currently in use, and to write the tracker name to the file using echo to switch to a different tracker. After the system starts, its default value is NOP, that is, no tracking action is made. After performing a tracking task, you can reset the tracker by writing a NOP to the file.
• available_tracers records the list of trackers currently compiled into the kernel, which can be viewed by cat, and contains trackers that correspond to the options that are activated in Figure 3. The tracker name used to write the Current_tracer file must be in the list of tracker names listed in the file.
• The trace file provides an interface to view the trace information obtained. You can view the file with commands such as cat to view the recorded kernel activity, or you can save its contents as a record file for later viewing.
• The tracing_enabled is used to control whether trackers in the Current_tracer can track the invocation of kernel functions. Writing 0 turns off the trace activity, and writing 1 activates the trace function, and its default value is 1.
• set_graph_function Sets the function to display the call relationship clearly, and the information structure is similar to the C language code, which makes it more intuitive to analyze the kernel's operational flow. used when using the Function_graph tracker; By default, a sequence of call relationships is generated for all functions, which can be written to specify functions that require special attention.
• The buffer_size_kb is used to set the size of the trace cache used by a single CPU. The tracker writes the tracked information to the cache, which is as large as the trace cache per CPU. The trace cache implementation is in the form of a ring buffer, and if too much information is traced, the old information is overwritten with the new trace information. Note that to change the value of the file, you need to set Current_tracer to NOP first.
• The tracing_on is used to control the pause of the trace. Sometimes when you observe certain events and want to temporarily turn off tracing, you can write 0 to the file to stop tracing, so that the newer part of the trace buffer is related to the event of interest; Write 1 can continue tracing.
• The available_filter_functions records the kernel functions that are currently traceable. For functions that are not listed in the file, their activities cannot be tracked.
• set_ftrace_filter and set_ftrace_notrace are used after the kernel is compiled with dynamic ftrace (with the Config_dynamic_ftrace option selected). The former is used to display the function specified to be traced, whereas the latter is used to specify a function that does not trace. If a function name appears in these two files at the same time, the execution status of this function is not tracked. These files also support simple forms of expressions with wildcard characters, so that you can specify multiple target functions at once using an expression, which is described in subsequent articles. Note that the function name to write to these two files must be visible in the file available_filter_functions. By default, all kernel functions can be traced, and the value of the file Set_ftrace_notrace is empty.

Back to top of page

Ftrace Tracking Device

Ftrace currently contains multiple trackers for tracking different types of information, such as process scheduling, interrupt shutdown, and so on. You can view the file Available_tracers get the list of trackers currently supported by the kernel. When compiling the kernel, you can also see the options that the kernel supports for trackers, as shown in Figure 3 earlier.

• The NOP Tracker does not track any kernel activity, writes NOP to the Current_tracer file to delete previously used trackers, and empties the trace information previously collected, that is, refreshing the trace file.
• the function tracker can track the execution of kernel functions, and you can specify which functions to trace by displaying the file set_ftrace_filter.
• The function_graph Tracker can display a function call graph similar to C source, which looks more intuitive; You can specify a function to generate a call flowchart by using the file set_grapch_function display.
• The Sched_switch Tracker can track process scheduling activity in the kernel.
• The Irqsoff Tracker and the Preemptoff Tracker track the code that closes the interrupt and the code that prohibits process preemption, respectively, and records the maximum length of time that is closed,Preemptirqsoff Trackers can be seen as a combination of them.

Ftrace also supports a number of other trackers, such as Initcall, Ksym_tracer, Mmiotrace, Sysprof, and so on. The Ftrace framework supports extensions to add new trackers. Readers can refer to the documents under the Documentation/trace directory in the kernel source package and the source files under Kernel/trace to understand the purpose of the other trackers and how to add new trackers.

Back to top of page

Summary

This series of articles describes the configuration and use of Ftrace. This article is the first part, the compilation configuration of Ftrace, the interface of User Configuration access Ftrace and ftrace data file, and describes the use of some of the trackers provided by Ftrace. Due to space limitations, this article does not describe the specific use of ftrace and how to interact with Ftrace in the code, which will be given in subsequent chapters of this series.

Resources

• View the kernel documentation documentation/trace/ftrace.txt learn about Ftrace.
• LWN on ftrace related articles.
• View the kernel documentation Readme to learn how to compile the kernel.
• See the documentation under the kernel documentation directory documentation/kbuild/for Kconfig and Kbuild.
• See Makefile files and kconfig files under kernel directory kernel/trace for ftrace related compilation options.
• Find out more about our most popular articles and tutorials in the DeveloperWorks Linux zone for more reference materials for Linux developers, including beginners for Linux.
• Check out all Linux tips and Linux tutorials on the developerWorks.

Debugging the Linux kernel with Ftrace, part 1th