Analysis on application of SYSTEMTAP monitoring in Linux system

Application scenario: One day, in our server PHP code path more than a log file, never noticed this log file, but the log file format is obviously not our generation, the format is relatively simple, even no function name,log level, is obviously the output of one of the Third-party libraries we use. What is the bad thing about that process calling a Third-party library? We are, of course, skeptical, and the semantics of log can be used as a preliminary judge of the process. But there is no evidence.

Some children's shoes said, to this suspicious process directly to execute lsof-p PID or file execution lsof files is OK, if this process opened this inexplicable file, it proved to be suspicious process wrote this log file. Unfortunately, this process is not daemon, and the execution time is particularly short, too late to lsof him.

This is the time when we systemtap turned out to be born. Systemtap, because of his customizable, is almost an omnipotent Swiss army knife. The first idea is to monitor the Sys_open, look at the end is that process in death, opened this inexplicable document.

Method One: Monitor Sys_open

We all know SYSTEMTAP can monitor system calls, open as a system call, we can naturally monitor, if the open file is exactly the file we want to track, we will PID, execname print out, the truth.

The code is as follows

Copy Code

function Is_open_creating:long (Flag:long) { Creat_flag = 4//0x4 = 00000100b if (Flag & Creat_flag) { Return 1 } return 0 } Probe begin { printf ("Monitor file Beginn") } Probe kernel.function ("Sys_open") { if (user_string ($filename) = = "/home/manu/shell/temp/abc.log") { creating = Is_open_creating ($mode); if (creating) { printf ("pid%ld (%s) Create the" file%sn, PID (), Execname (), user_string ($filename)); } Else { printf ("pid%ld (%s) open" file%s n ", PID (), Execname (), user_string ($filename)); } } }

OK, let's start monitoring to see if we can catch the troublemakers.

The code is as follows	Copy Code
root@manu:~/code/systemtap# root@manu:~/code/systemtap# STAP FILE_MONITOR.STP Monitor file begin

We created this/home/manu/shell/temp/abc.log in another terminal L, Echo.

The code is as follows	Copy Code
root@manu:~/code/shell/temp# Echo ABEFDF >/home/manu/code/shell/temp/abc.log root@manu:~/code/shell/temp#

We've seen Stap capture this event:

The code is as follows	Copy Code
root@manu:~/code/systemtap# STAP FILE_MONITOR.STP Monitor file begin PID 3024 (bash) Create the File/home/manu/code/shell/temp/abc.log

STAP caught the process named Bash,pid 3024 to create the file. So far, all is well, but there is a fatal flaw in this approach. FileName is the name of the file entered when the process calls the system call open, it may enter a full path/home/manu/shell/temp/abc.log, or it may enter a relative path, and if the relative path is entered, our STAP cannot capture the event.

For example, once again we want Abc.log to append:

The code is as follows	Copy Code
root@manu:~/code/shell/temp# echo "second line" >> Abc.log root@manu:~/code/shell/temp# Cat Abc.log Abefdf Second line root@manu:~/code/shell/temp#

There is no STAP on the other end and no events are detected.
This approach is flawed because we are not able to assume the absolute path or relative path that the process enters.

Method Two: The inode of the monitoring file

The name of the file may be different, such as the current path is/home/manu/shell/temp/, the following is represented by the same file, which brings difficulties to the above method.

Abc.log
./abc.log
.. /temp/abc.log
/home/manu/shell/temp/abc.log
.....
If our files are on disk, then as long as there are three elements (main device number, sub device, Inode), only one file is identified. We still monitor the abc.log. For my file in/dev/sda6, the corresponding main device number and the secondary device number are (0x8,0x6), abc.log corresponding Inode is:

The code is as follows	Copy Code
361way:/opt # Ll-ali Abc.log 2099351-rwxr-xr-x 1 root 17623 Sep 2 22:55 abc.log

Our ternary group is (0x08,0x06,2099351)
Here's our monitor script INODE_MONITOR.STP

The code is as follows	Copy Code
Probe begin { printf ("Watch Inode%d%d%ld beginn", $1,$2,$3) } Probe Vfs.write { if (dev = = Mkdev ($1,$2) # Major/minor Device && ino = $) printf ("%s (%d)%s 0x%x/%un", Execname (), PID (), Probefunc (), Dev, ino) }

And then we have STAP to monitor the file for this inode.

The code is as follows	Copy Code
root@manu:~/code/systemtap# Stap inode_monitor.stp 0x8 0x6 2099351 Watch Inode 8 6 2099351 begin

At the beginning of our experiment, we wrote echo on the shell Terminal Abc.log, writing a test.sh script to write this abc.log, the experiment is as follows

The code is as follows	Copy Code
root@manu:~/code/shell/temp# Echo ABEFDF >abc.log root@manu:~/code/shell/temp# echo "second line" >/abc.log root@manu:~/code/shell/temp# Cat test.sh #!/bin/sh echo "Third line" >>/abc.log root@manu:~/code/shell/temp#./test.sh

The output of the STAP monitoring script is as follows:

The code is as follows	Copy Code
root@manu:~/code/systemtap# Stap inode_monitor.stp 0x8 0x6 2099351 Watch Inode 8 6 2099351 begin Bash (3024) vfs_write 0x800006/2099351 Bash (3024) vfs_write 0x800006/2099351 Test.sh (9484) Vfs_write 0x800006/2099351

We see all three of these events captured. We have completed our monitoring of this file to determine exactly what the process is writing to this log target.

Systemtap as a tool for dynamic monitoring and tracking of the Linux kernel, its functionality can be explored more. See the IBM Community or SYSTEMTAP official website for more information. You can also learn about DTrace, ProbeVue (both on UNIX platforms) and Fanotify (the next generation of inotify file monitoring) similar tools.

Other performance Tools Oprofile, Valgrind, Perf, Redhat MGR, and so on back to sum up learning