Ftrace use "Turn"

Transferred from: http://blog.csdn.net/cybertan/article/details/8258394

This article explains how to set up Ftrace and is able to understand how to trace functions. It should is useful for the current kernel developers and device driver developers who want to debug kernel issues, and also F Or students who is keen to pursue a Linux systems programming career.

This article explores how to build ftrace and understand how to track functions. Ftrace should be useful for kernel developers and device driver developers when debugging kernel issues. This article is also worth learning for enthusiasts who are eager to explore Linux system programming.

Ftrace (Function Tracer) is the "Swiss Army Knife" of kernel tracing. It's a tracing mechanism built right into the Linux kernel. It has the capability to see exactly what's happening in the kernel, and debug it. Ftrace is more than a mere function tracer, and have a wide variety of tracing abilities to debug and analyse a number of I Ssues like latency, unexpected code paths, performance issues, etc. It can also be used as a good learning tool.

Ftrace is introduced in kernel 2.6.27 by Steven rostedy and Ingo Molnar. It comes with its own ring buffer for storing trace data, and uses the GCC profiling mechanism.

Ftrace (function Tracking) is the "Swiss Army Knife" traced by the kernel. It is a tracking mechanism built into the Linux kernel. It can go deep into the kernel to find out what's going on inside and debug it. Ftrace is not just a function tracking tool, it's a powerful tracking capability, but also can debug and analyze a lot of problems such as latency, unexpected code paths, performance issues, and so on. It's also a great learning tool.

Ftrace was introduced by Steven Rostedy and Ingo Molnar in the kernel 2.6.27 release. It has its own ring buffer for storing trace data and uses the GCC configuration mechanism.

Prerequisites Preparation Knowledge

You need a 32-bit or 64-bit Linux machine with a kernel development environment, and as new a kernel as possible (the Newe R the kernel, the more of the tracing options you get). I use a Fedora Core (x86_64) machine in my environment, but any distribution would suffice.

You need a 32-bit or 64-bit Linux machine with a kernel development environment, the newer the kernel, the better (the newer the kernel, the more tracking options). I am a Fedora13-based kernel (x86_64) machine, but other versions also apply.

Setting Up Ftrace Settings ftrace

Debugfs needs to is set up-to-run on the machine you want to use Ftrace on. If you are unaware the to set up Debugfs, do refer to my Debugfs article from last month.

Debugfs should has been mounted on/sys/kernel/debugfs, and if tracing is enabled, you should being able to see a directory Called tracing under Debugfs. If Debugfs isn't mounted, you can issue the following command:

Using ftrace requires that you have Debugfs configured on your machine. If you don't know how to configure Debugfs, please refer to my article last month.

Debugfs should be mounted in/sys/kernel/debugfs, if the tracking option is enabled, you should be able to see a directory called tracing under Debugfs. If the Debugfs is not mounted, please do the following:

# mount-t Debugfs Nodev/sys/kernel/debug

If you is unable to see the tracing subdirectory, you'll need to enable tracing in the kernel configuration, and Recomp Ile it. Look for the following options to is enabled in the kernel configuration path (refer to Figure 1):

Kernel Hacking-tracers
1. Kernel Function Tracer (function_tracer)
2. Kernel Function Graph Tracer (function_graph_tracer)
3. Enable/disable Ftrace Dynamically (Dynamic_ftrace)
4. Trace Max Stack (stack_tracer)

If you do not see the tracing subdirectory, you should enable the relevant options on the kernel configuration and then recompile the kernel. Please find the options shown in 1 in your kernel configuration to enable them:

Kernel Hacking-tracers
1. Kernel Function Tracer (function_tracer)
2. Kernel Function Graph Tracer (function_graph_tracer)
3. Enable/disable Ftrace Dynamically (Dynamic_ftrace)
4. Trace Max Stack (stack_tracer)

Figure 1:kernel configurationoptions for tracing diagram 1:tracing kernel configuration options

Depending on your architecture, a few more tracers can is enabled during compilation, as per requirements. The listed tracers is for debugging. Once the kernel compilation is complete, and you are booted to the new kernel, tracing can initiated.

Depending on your schema, some of the other options may be automatically enabled depending on your dependencies when selecting the options above. The options listed above are primarily used for tracking purposes. After the kernel is compiled, you only need to restart the machine, the tracing function can be used.

Tracing tracking

Files in the tracing directory (/sys/kernel/debug/tracing) control the tracing ability (refer to Figure 2 for a list of fi Les). A few files could is different, depending upon what tracers you selected during kernel configuration. You can obtain information in these files from the <kernel source>/documentation/tracing directory.

The files in the tracing directory (/sys/kernel/debug/tracing) (shown in 2) control the ability to track. Depending on your options in the kernel configuration, the files listed here may be slightly different. You can find the information for these files in the/documentation/trace[1] directory under the kernel source code directory.

Figure 2:tracing Files Fig. 2: File for tracking

Let ' s explore a few of the important ones:

Available_tracers:this shows what tracers is compiled to trace the system.
Current_tracer:displays What tracer is currently enabled. Can is changed by echoing a new tracer into it.
Tracing_enabled:lets you enable or disable the current tracing.
Trace:actual trace output.
Set_ftrace_pid:sets the PID of the process for which trace needs to be performed.

To find out the available tracers, just cat the Available_tracers file. Tracers in the space-separated output Include:nop (not a tracer, which is set by default); function (function tracer); Function_graph (function Graph tracer), etc:

Let's look at a few important documents:

Available_tracers: This indicates which trackers are compiled in the system.
Current_tracer: This indicates which tracker is currently enabled. You can change the value by entering a new tracker with Echo to the table.
Tracing_enabled: Allows you to enable or disable the current tracking feature
Trace: The actual trace output.
Set_ftrace_pid: Sets the PID of the process that the trace is acting on.

To find which trackers are available, you can perform a cat operation on the Available_tracers file. The tracker separated from the output space is: NOP (it is not a tracker, is a value of the default setting), a function (function tracker), a function graph (function graph tracker), and so on, as follows:

# Cat Available_tracers

Blk function_graph mmiotrace wakeup_rt wakeup irqsoff function Sched_switch NOP

Once you identify the tracer, enable it (Ftrace takes-one tracer at a time):

When you know which tracker you need to use, enable it (Ftrace can only open one tracker at a time):

# cat Current_tracer # #to See what Tracer are currently in use.

# echo function > Current_tracer # #select a particular tracer.

# cat Current_tracer # #check Whether we got what we wanted.

# cat Current_tracer # #查看当前在用哪个跟踪器.

# echo function > Current_tracer # #选择一个特定的跟踪器.

# cat Current_tracer # #检查是否是你所设置的跟踪器.

To start tracing, use the following commands:

Use the following command to start the trace:

# echo 1 > tracing_enabled # #initiate tracing

# cat trace >/tmp/trace.txt # #save The contents of the trace to a temporary file.

# echo 0 > tracing_enabled # #disable tracing

# cat/tmp/trace.txt # #to See the output of the the trace file.

# echo 1 > Tracing_enabled # #初始化跟踪.

# cat trace >/tmp/trace.txt # #将跟踪文件保存到一个临时文件.

# echo 0 > Tracing_enabled # #禁用跟踪功能

# cat/tmp/trace.txt # #查看trace文件的输出.

The trace output is now in the Trace.txt file. A sample output of a function trace obtained with the above commands was shown in Figure 3.

The input to the trace file is now in the Trace.txt file. A sample of the function trace obtained by the above operation is shown in output 3.

Figure 3:sample Trace output Fig. 3: Sample Trace Outputs

The English source text of this article is http://www.linuxforu.com/2010/11/kernel-tracing-with-ftrace-part-1/, and it is attached to the following articles.

Let's begin with tracer options. The output of the tracing can is controlled by a file called Trace_options. Various fields can is enabled and disabled by updating options in the File/sys/kernel/debug/tracing/trace_options. A sample of Trace_options can viewed in Figure 1.

Let's start with the tracer option. The input of the tracing can be controlled by a file called Trace_options. You can enable or disable various domains by updating the options for the/sys/kernel/debug/tracing/trace_options file. An example of Trace_options is shown in 1.

Figure 1:trace Options Chart 1:trace option

To disable a tracing option, a "no" keyword needs to being added to the start of the line. For example, echo NOTRACE_PRINTK > Trace_options. (Remember not to has a space between no and the option.) To enable a trace again, you could use, for instance, echo TRACE_PRINTK > Trace_options.

To disable a tracking option, simply add a "no" to the beginning of the line. For example, echo NOTRACE_PRINTK > Trace_options. (There is no space between the No and options.) To enable a tracking option again, you can do this: Echo TRACE_PRINTK > Trace_options.

Ftrace for a specific processftrace special process

Ftrace allows-perform tracing even for a specific process. In the/sys/kernel/debug/tracing directory, the file Set_ftrace_pid needs-to is updated with the PID of the process you WA NT to is traced. The Traceprocess.sh sample script below shows how to capture the PID on-the-go, and enable tracing.

Ftrace allows you to track a particular process. In the/sys/kernel/debug/tracing directory, the value of the file Set_ftrace_pid is updated to the PID of the process you want to track. The following traceprocess.sh sample script shows you how to crawl the PID of the currently running process and track it accordingly.

View Code

You can refine it with your own innovations. Run it with the command whose process your want to trace as the argument, as shown in Figure 2, where we traced the LS comm and.

You can also improve the above script as needed. Run the above script on the command line and follow the LS command as a parameter in the process name that you want to track, as shown in 2.

Figure 2:executing traceprocess.sh and viewing trace output fig. 2: Performing traceprocess.sh and viewing trace outputs

Once tracing is complete, you need to clear the Set_ftrace_pid file, for which you can use the following command:

When the trace is complete, you need to clear the Set_ftrace_pid file, please use the following command:

:> Set_ftrace_pid

Function Graph Tracer Graph Tracker

The function Graph tracer tracks the entry and exit of a function, and is quite useful to track its execution time. Functions with a duration of over ten microseconds is marked with a "+", and those over the microseconds with "!". To enable the function graph tracer, use echo function_graph > Current_tracer. The sample output is as shown in Figure 3.

The function Graph tracker tracks the entry and exit of a function, which is useful for tracking its execution time. function execution time is more than 10 microseconds marked with a "+" sign, more than 1000 microseconds marked as one "! No. The function graph Tracker can be enabled via echo function_graph > Current_tracer. Sample input 3 shows.

Figure 3:trace Output of function graph Tracer fig. 3: Function graph Tracker Trace outputs

There is a lot of tracers; The entire list is in Linux/documentation/trace/ftrace.txt. The tracers is enabled or disabled by echoing the tracer name into the Current_tracer file.

There are many trackers, and all the lists are found in the Linux/documentation/trace/ftrace.txt file. You can enable or disable the Tracker by echo the tracker's name to the Current_tracer file.

Dynamic tracing Tracking

We can easily get inundated with the amount of data, the function tracer throws at us. There is a dynamic just the functions we need, and eliminate those so we don ' t need:to specify them in th E file Set_ftrace_filter. (First find the function (s) want, from the Available_filter_functions file.) See Figure 4 for an example of dynamic tracing.

We are easily overwhelmed by the amount of data that the function tracker throws at us. There is a dynamic way to filter out the functions we need to exclude those we do not need: indicated in the file set_ftrace_filter. (First find the function you need from the Available_filter_functions file.) Figure 4 is an example of a dynamic tracking.

Figure 4:listing Filter functions, using with Wild-cards fig. 4: List the filtered functions using wildcards

As can see, you can even use wild-cards for the functions ' names. I used all the VMALLOC_ functions, and set them With:echo vmalloc_* > Set_ftrace_filter.

As you can see, you could even use wildcard characters for the name of the function. I need all the VMALLOC_ functions to be set up with echo vmalloc_* > Set_ftrace_filter.

Event Tracing Events Tracking

Tracing can also is triggered when particular events happen on the system. Available system events is found in the file available_events:

You can also turn on tracing when a system-specific event is triggered. All available system events can be found in the Available_events file:

[email protected] tracing]# Cat available_events | Head-10

Kvmmmu:kvm_mmu_pagetable_walk

Kvmmmu:kvm_mmu_paging_element

Kvmmmu:kvm_mmu_set_accessed_bit

Kvmmmu:kvm_mmu_set_dirty_bit

Kvmmmu:kvm_mmu_walker_error

Kvmmmu:kvm_mmu_get_page

Kvmmmu:kvm_mmu_sync_page

Kvmmmu:kvm_mmu_unsync_page

Kvmmmu:kvm_mmu_prepare_zap_page

Kvm:kvm_entry

For example, to enable a event, you would use:echo Sys_enter_nice >> set_event (Note this you append the event Nam E to the file, using the >> append redirector, and not >). To disable an event, precede the event name with a "!": Echo '!sys_enter_nice ' >> set_event. See Figure 5 for a sample event tracing scenario. The available events is listed in the events directory as well.

For example, to enable an event, you need: Echo Sys_enter_nice >> set_event (Note that you are appending the name of the event to the file, using the >> append redirector, not >). To disable an event, you need to precede the name with a "!" Number: Echo '!sys_enter_nice ' >> set_event. Figure 5 is an example of an event tracking scenario. Similarly, the available events are listed in the event directory.

Figure 5:available Tracing events, setting and ' unsetting ' them. Figure 5: Available trace events, enabling and disabling them

For further details on event tracing, read the file documents/trace/events.txt in the kernel directory.

For more details on event tracking, please read the Documents/trace/events.txt file in the kernel directory.

Trace-cmd and Kernelsharktrace-cmd and Kernelshark

Trace-cmd, introduced by Steven Rostedt in He July post to the LKML, makes it easy to manipulate the tracer. Follow these steps to get the latest version, including the GUI tool Kernelshark, installed on your system:

Trace-cmd was made by Steven Rostedt in the 2009 on LKML, which makes it easier to operate trackers. The following steps are to get the latest version and install it on your system, including its GUI tools Kernelshark.

wget http://ftp.be.debian.org/pub/linux/analysis/trace-cmd/ trace-cmd-1.0.5.tar.gz [2]

TAR-ZXVF trace-cmd-1.0.5.tar.gz

CD trace-cmd*

Make

Make GUI # compiles GUI tools (Kernelshark) [3]

Make install

Make Install_gui # installs GUI tools

With Trace-cmd, tracing becomes a breeze (see Figure 6 for sample usage):

With trace-cmd, tracking will become a piece of cake (see example usage in Figure 6):

Trace-cmd List # #to See available events

Trace-cmd record-e syscalls ls # #Initiate tracing on the syscall ' ls '

# # (A file called Trace.dat gets created in the current directory.)

Trace-cmd Report # # Displays the report from Trace.dat

Figure 6:using Trace-cmd for recording and reporting fig. 6: Record and report with Trace-cmd

Kernelshark, installed by the make Install_gui step above, can is used to analyse the trace data in the file Trace.dat, as shown in Figure 7.

The Kernelshark that is installed through the make Install_gui command above can be used to analyze trace data in the Trace.dat file, as shown in 7. [4]

Figure 7:analysing Trace-cmd output with Kernelshark fig. 7: Analyzing trace-cmd outputs with Kernelshark

Ftrace use "Turn"