Linux system fault checking artifact Sysdig installation Instructions

Sysdig listens at the operating system level and captures system activities such as system calls and system events, which makes it look very much like a system-oriented tcpdump or Wireshark. If you plan to troubleshoot the system, then Sysdig will be a handy tool to solve your problems.

On Linux, you can use the following command to install Sysdig:

Curl-s Https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash

This will install the Sysdig to the RPM or Deb Linux system.

Capturing system activity

Real-time capture, resulting in print to standard output:

Sysdig

Save capture results to file System.scap for later analysis:

Sysdig-w System.scap

Captures the specified number of events by 200 and saves to a file:

Sysdig-n 200-w System.scap

To read a captured file:

Sysdig-r System.scap

Capture result Interpretation

(1) (2) (3) (4) (5) (6) (7) (8)
1 10:54:50.462463956 0 sysdig (29043) > Sysdigevent event_type=1 event_data=0
2 10:54:50.462603110 0 sysdig (29043) > Sysdigevent event_type=1 event_data=0
3 10:54:50.462729565 0 Sysdig (29043) > Sysdigevent event_type=1 event_data=0
4 10:54:50.462859521 0 Sysdig (29043) > Sysdigevent event_type=1 event_data=0
5 10:54:50.463206317 0 Sysdig (29043) > Switch next=0 pgft_maj=0 pgft_min=1790 vm_size=35748 vm_rss=7164 vm_swap=0
6 10:54:50.464246835 0 <NA> (0) > Switch next=7 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0
7 10:54:50.464249707 2 <NA> (0) > Switch next=8374 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0
8 10:54:50.464255940 0 <NA> (7) > Switch next=0 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0
9 10:54:50.464264256 2 <NA> (8374) > Switch next=0 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0
10:54:50.464358113 2 <NA> (0) > Switch next=854 (mlnet) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0
One 10:54:50.464370099 2 mlnet (854) < poll res=0 fds=
10:54:50.464378193 2 mlnet (854) > Poll fds= timeout=5
10:54:50.464385400 2 mlnet (854) > Switch next=0 pgft_maj=216 pgft_min=3386 vm_size=162608 vm_rss=12196 vm_swap=2716
10:54:50.464950541 0 <NA> (0) > Switch next=1105 (memcached) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0
10:54:50.464954692 0 memcached (1105) < epoll_wait res=0
10:54:50.464976007 0 memcached (1105) > Epoll_wait maxevents=32
10:54:50.464984030 0 memcached (1105) > Switch next=0 pgft_maj=3 pgft_min=247 vm_size=327412 vm_rss=1860 vm_swap=468
10:54:50.465256687 2 <NA> (0) > Switch next=2181 (plugin-containe) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_ Swap=0
10:54:50.465261465 2 Plugin-containe (2181) < poll res=0 fds=
10:54:50.465297692 2 plugin-containe (2181) > Getrlimit resource=3 (rlimit_stack)

The results captured by Sysdig as shown above, each column means:

Event Number
Time stamp
CPU number
Process Name
Thread ID
Event Direction,> for entry event,< for Exit event
Event types, such as open, read, and so on
Event argument list

Filter Capture Results

By default, Sysdig captures so much information that we need to find the information we're interested in, which requires a filter like grep.

Filter by field Category:

Sysdig-r System.scap Proc.name=sysdig

This command filters out system events with process name Sysdig, and the result is:

1 10:54:50.462463956 0 sysdig (29043) > Sysdigevent event_type=1 event_data=0
2 10:54:50.462603110 0 sysdig (29043) > Sysdigevent event_type=1 event_data=0
3 10:54:50.462729565 0 Sysdig (29043) > Sysdigevent event_type=1 event_data=0
4 10:54:50.462859521 0 Sysdig (29043) > Sysdigevent event_type=1 event_data=0
5 10:54:50.463206317 0 Sysdig (29043) > Switch next=0 pgft_maj=0 pgft_min=1790 vm_size=35748 vm_rss=7164 vm_swap=0

Sysdig provides field categories including FD, process, evt, user, group, and Syslog, which can be queried by sysdig-l.

In addition to =, the Sysdig filter expression also supports comparison operators such as!=, <, <=, >, >=, and contains.

You can also use Boolean operators such as and, or, not. For example:

Sysdig-r System.scap Proc.name=sysdig and Evt.type=switch

Chisels

In Sysdig, chisels is a script written in Lua that can be used to extend sysdig filtering capabilities.

For example, we want to see the most frequently read and write disk files process, you can use topprocs_file this chisels:

Sysdig-c Topprocs_file

The results are:

Bytes Process
------------------------------
448.36KB Mozstorage
220.38KB Perl
1.69KB Tmux
1.62KB SH
1.59KB Xorg
1.30KB URXVTD

More chisels, can be understood through SYSDIG-CL. Of course, if you are familiar with Lua, you can also write your own chisels.