Use ftrace to debug the kernel

Ftrace is used to help developers understand the Linux kernel running track for fault debugging or performance analysis. It is in kernel 2.6.27
Since then, a new tracer has been added. Up to 2.6.30, there have been at least 10 tracer types. For more information about tracer, see
/Documentation/trace/ftrace.txt.

This article describes how to use ftrace:
To use ftrace, you must first add support for it in the kernel, that is, re-compile a new kernel.
Kernel hacking
Tracers
Kernal Function Tracer
Kernal Function Graph Tracer
.......
Yi
Compile these options into the kernel. In this way, you can enjoy the convenience that ftrace brings to your new compiled kernel. Ftrace is output from debugfs.
Port to read the kernel information. When debugfs is configured to the kernel, A/sys/kernel/debug directory is created. The kernel information we want to view must be obtained from this directory.
.
First, debugfs must be mounted. Run the following command to mount
Sudo mount-t debugfs nodev/sys/kernel/debug/
You can also add it to the/etc/fstab file.

Next, to facilitate access, establish a soft link and link the/sys/kernel/debug directory to the/debug directory (the/debug directory already exists in 2.6.30, earlier kernel may not exist). If not, you can create it manually.
Wuting @ wuting-desktop :~ $ Sudo ln-s/sys/kernel/debug/tracing // debug/
Wuting @ wuting-desktop :~ $ Ls/debug/
Tracing
Run the ls command to view the following files in the tracing directory:
Wuting @ wuting-desktop :~ $ Ls/debug/tracing
Available_events events saved_cmdlines stack_max_size trace_stat
Available_filter_functions failures set_event stack_trace tracing_cpumask
Available_tracers options set_ftrace_filter trace tracing_enabled
Buffer_size_kb per_cpu set_ftrace_notrace trace_marker tracing_max_latency
Current_tracer printk_formats set_ftrace_pid trace_options tracing_on
Dyn_ftrace_total_info readme set_graph_function trace_pipe tracing_thresh
In addition to the directory (green), all files here can be written using the echo command, and read using the cat command. In this way, we can use echo and cat to configure ftrace.

Finally, an example is provided to illustrate the specific usage of ftrace.
Echo function_graph>/debug/tracing/current_tracer // configure to use function graph tracer to track function calls.
Echo 10000>/debug/tracing/buffer_size_kb // configure the ring buffer size to bytes kb.
Echo 1>/debug/tracing/tracing_enabled // enable trace
/* Execute the program to be tracked, such as LS ,*/
Ls-l
Echo 0>/debug/tracing/tracing_enabled // disable trace
CAT/debug/tracing/Trace> trace.txt // read the trace information and redirect it to a file for ease of viewing

Open trace.txt and we can see that the call relationship between kernel functions is displayed in a graphical form. The first column indicates the CPU on which the function is executed, the second column indicates the time used for function execution, the plus sign (+) indicates a delay, and the third column indicates the call relationship between functions.

# Tracer: function_graph
#
# CPU duration function CILS
# |
0) + 55.570 us |}
0) 0.675 us | strncpy ();
0) + 58.825 us |}
0) + 61.605 us |}
0) 0.465 us | strcmp ();
0) | kallsyms_lookup (){
0) | is_ksym_addr (){
0) 0.455 USD | in_gate_area_no_task ();
0) 1.375 us |}
0) | module_address_lookup (){
0) + 55.645 us | get_ksymbol ();
0) 0.685 us | strncpy ();
0) + 58.525 us |}
0) + 61.315 us |}
0) 0.465 us | strcmp ();
...................
To sum up, the use of ftrace is:
● Select a tracer
● Enable Tracer
● Execute the application to be traced
● Disable ftrace
● View the trace file