Sysdig: a powerful tool for system troubleshooting

Sysdig: a powerful tool for system troubleshooting

Sysdig monitors the operating system and captures system activities such as system calls and system events, making it look like a system-oriented tcpdump or Wireshark. If you plan to troubleshoot system exceptions, Sysdig will be a handy tool to solve the problem.

Sysdig: A Tool for Linux Server monitoring and troubleshooting

On Linux, run the following command to install Sysdig:

curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash

This will install Sysdig to the rpm or deb Linux system.

Capture System activities

Real-time capture and the result is printed to the standard output:

sysdig

Save the captured results to the file system. scap for later analysis:

sysdig -w system.scap

Capture 200 of the specified events and save them to the file:

sysdig -n 200 -w system.scap

Read captured files:

sysdig -r system.scap

Capture result explanation

(1)      (2)        (3) (4)     (5)  (6)    (7)            (8)1 10:54:50.462463956 0 sysdig (29043) > sysdigevent event_type=1 event_data=0 2 10:54:50.462603110 0 sysdig (29043) > sysdigevent event_type=1 event_data=0 3 10:54:50.462729565 0 sysdig (29043) > sysdigevent event_type=1 event_data=0 4 10:54:50.462859521 0 sysdig (29043) > sysdigevent event_type=1 event_data=0 5 10:54:50.463206317 0 sysdig (29043) > switch next=0 pgft_maj=0 pgft_min=1790 vm_size=35748 vm_rss=7164 vm_swap=0 6 10:54:50.464246835 0 <NA> (0) > switch next=7 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 7 10:54:50.464249707 2 <NA> (0) > switch next=8374 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 8 10:54:50.464255940 0 <NA> (7) > switch next=0 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 9 10:54:50.464264256 2 <NA> (8374) > switch next=0 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 10 10:54:50.464358113 2 <NA> (0) > switch next=854(mlnet) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 11 10:54:50.464370099 2 mlnet (854) < poll res=0 fds= 12 10:54:50.464378193 2 mlnet (854) > poll fds= timeout=5 13 10:54:50.464385400 2 mlnet (854) > switch next=0 pgft_maj=216 pgft_min=3386 vm_size=162608 vm_rss=12196 vm_swap=2716 14 10:54:50.464950541 0 <NA> (0) > switch next=1105(memcached) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 15 10:54:50.464954692 0 memcached (1105) < epoll_wait res=0 16 10:54:50.464976007 0 memcached (1105) > epoll_wait maxevents=32 17 10:54:50.464984030 0 memcached (1105) > switch next=0 pgft_maj=3 pgft_min=247 vm_size=327412 vm_rss=1860 vm_swap=468 18 10:54:50.465256687 2 <NA> (0) > switch next=2181(plugin-containe) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 19 10:54:50.465261465 2 plugin-containe (2181) < poll res=0 fds= 20 10:54:50.465297692 2 plugin-containe (2181) > getrlimit resource=3(RLIMIT_STACK) 

The results captured by Sysdig are as follows:

1. Event ID
2. Timestamp
3. CPU ID
4. Process name
5. Thread ID
6. Event direction,> to enter the event, <to exit the event
7. Event type, such as open and read
8. Event parameter list

Filter capture results

By default, Sysdig captures a large amount of information from which to find the information we are interested in. This requires a filtering function similar to grep.

Filter by field category:

sysdig -r system.scap proc.name=sysdig

This command filters out the system event with the process name sysdig. The result is:

1 10:54:50.462463956 0 sysdig (29043) > sysdigevent event_type=1 event_data=0 2 10:54:50.462603110 0 sysdig (29043) > sysdigevent event_type=1 event_data=0 3 10:54:50.462729565 0 sysdig (29043) > sysdigevent event_type=1 event_data=0 4 10:54:50.462859521 0 sysdig (29043) > sysdigevent event_type=1 event_data=0 5 10:54:50.463206317 0 sysdig (29043) > switch next=0 pgft_maj=0 pgft_min=1790 vm_size=35748 vm_rss=7164 vm_swap=0 

Sysdig provides fields including fd, process, evt, user, group, and syslog.sysdig -lQuery.

In addition to =, The Sysdig filter expression also supports! Comparison operators such as =, <, <=,>,> =, and contains.

You can also use boolean operators such as and, or, and not. For example:

sysdig -r system.scap proc.name=sysdig and evt.type=switch

Chisels

In Sysdig, chisels is a script written by Lua and can be used to extend the filtering function of Sysdig.

For example, if you want to read and write the disk files frequently, you can use the topprocs_file chisels:

sysdig -c topprocs_file

Result:

Bytes     Process   ------------------------------448.36KB  mozStorage220.38KB  perl1.69KB    tmux1.62KB    sh1.59KB    Xorg1.30KB    urxvtd

For more chisels, you can usesysdig -clLearn more. Of course, if you are familiar with Lua, you can also write your own chisels.

This article permanently updates the link address: