Linux BPF/BCC for Oracle tracing

Luca Canali on 2016

Topic: In this post you'll find a short discussion and pointers to the code of a few sample scripts that I had written using L Inux BPF/BCC and Uprobes for Oracle tracing.

Previous Work and motivations

Tools forDynamic Tracingis very useful for troubleshooting and internals investigations ofOracleWorkloads. Dynamic tracing probes on the os/Kernel, can is used to measure the details for I/O latency for example. Moreover probes on the OracleUserspaceFunctions can be is used to complement Oracle instrumentation and probe deeper into the internals of the Oracle engine, WH En needed. For example in a work-in-collaboration with Frits Hoogland(link is external)We have investigated how to link Oracle wait event interface with tools able to probe the userspace, such as GDB and Dynam IC tracing tools as DTrace, SystemTap and Perf. More details and examples on this topic can is found in previous Work:modern Linux Tools for Oracle Troubleshooting (pres Entation at Ukoug Tech14), life of an Oracle I/O: tracing Logical and physical I/O with SystemTap(link is external), Linux Perf probes for Oracle tracing(link is external), Flame Graphs for Oracle(link is external).

What ' s new with BPF/BCC

BPF together with He frontend bcc(link is external) are new and exciting TEC Hnologies in the Linux dynamic tracing ecosystem. If you is not a familiar with them, you can get up to speed using the excellent material in Brendan Gregg ' s blog(link i s external). While the tools is still in development as I write this, a large amount of important features is already available start ing from kernel 4.5, notably the support for Uprobes that I had used in the scripts discussed later in this post.

I have started experimenting with porting a few probes for the Oracle userspace that I had written using Systemta P and discussed in the this blog. While the language/syntax are completely different between SystemTap and BPF/BCC, porting The probes has turned OU T to be relatively straightforward. The work have been made substantially easier by the availability of a large selection of well-written and document Ed scripts in the Tools directory of BCC(link is external). At present writing BPF/BCC scripts are a mixture of coding in Python and C, which feels to me both familiar and at the same Time strangely low-level compared to the more mature environments for example of DTrace and SystemTap.

The example scripts that I had written using BPF/BCC is on Github(link is external). Here's a list with comments:

Script Name	Short description
ora_sqlparse_trace.py (link is external)	Tracing of Oracle SQL parsing. This script traces SQL hard parsing in Oracle binaries hooking on the Oracle function ' OPIPRS ' and reads from function arg Uments (CPU registers) and from process memory.
ora_wait_trace.py (link is external)	Tracing of Oracle wait events. This script traces Oracle sessions by hooking on the functions "KSKTHEWT" and "Kews_update_wait_time" and reads from FUNCT Ion arguments (CPU registers).
ora_logicalio_histogram.py (link is external)	Logical IO Latency histograms. This script measures the latency between call and return time for the Oracle function "KCBGTCR", which was an important par T of the logical IO processing for consistent reads.
ora_wait_histogram.py (link is external)	Wait Event Latency histograms. This script traces Oracle sessions by hooking on the functions "KSKTHEWT" and "Kews_update_wait_time" and reads from FUNCT Ion arguments (CPU registers). BPF computes the latency histogram for the wait events and the script prints the values on stdout.

An example of the usage of ora_wait_histogram.py to measure and display wait event latency:

# Stdbuf-ol/ora_wait_histogram.py 10|sed-e ' s/event# =/event#=/g '-F eventsname.sed

Start tracing Oracle Wait Events ... Hit Ctrl-c to end.

EVENT=DB file Sequential Read

Wait time, Microsec:count distribution

0-1:0 | |

2-3:0 | |

4-7:0 | |

8-15:0 | |

31:0 | |

63:0 | |

127:25 | |

255:24,521 |******************** |

511:46,788 |****************************************|

1023:12,169 |********** |

2047:1132, 1024x768 | |

2048, 4,095:660 | |

4096, 8,191:248 | |

8192, 16,383:29 | |

Latency heat maps to display histograms collected with BPF/BCC

Pylatencymap(link is external)is a command-line tool for visualizinglatency histograms using heat mapsOn terminal graphics, using ANSI escape codes. Pylatencymap can used to investigate I/O performance for random I/O, especially suited for the cases of multiple modes Of response time from the storage (SSD cache, HDD, latency outliers). The original idea for Pylatencymap comes from the work of Brendan Gregg(link is external)On latency heat maps. I have added to the Pylatencymap repository(link is external)A modified version of the biolatency.py(link is external)Script to measure I/O latency histograms:this is the link to the script pylatencymap_biolatency.py(link is external)Example of heat maps generated with Pylatencymap with the script example11_bpf-bcc_blockio_latency.sh(link is external):

The test workload has been generated using Kevin Closson's slob(link is external). Additional references with a more detailed discussion of the topic of testing Oracle I/O with slob and measuring latency H Eat maps are:

Heat Map Visualization of I/O Latency with SystemTap and Pylatencymap(link was external) and Oralatencymap v1.1 an D testing I/O with slob 2(link is external).

Notes on the test environment

The scripts discussed in this post has been developed on Fedora (Alpha) running Linux kernel version 4.6 (usi ng the Rawhide kernel) and has been tested on workloads generated using Oracle version 11.2.0.4. This isn't an oracle-supported configuration and the scripts be intended mainly as a demonstration of the technology an D for learning purposes.

Here is some pointers on the steps I used to setup a lab environment for testing:

• Download Fedora Server from https://getfedora.org/en/server/(link is external) (version of Beta, as I write this) .
• Installed Fedora, for example on a VMS with VirtualBox(link is external)
• Run "sudo DNF update" to get the latest kernel and packages
• Get kernel-devel with "sudo DNF install Kernel-devel"
• Follow the installation instructions for BCC in Fedora from Https://github.com/iovisor/bcc/blob/master/INSTALL.md (link is external)

Note in particular the step for configuring the RawhideKernel, probably a good choice when testing BPF, as new features is being added on a regular basis as I write this:# sudo DNF Config-manager--add-repo=http://alt.fedoraproject.org/pub/alt/rawhide-kernel-nodebug/fedora-rawhi ...(link is external)# sudo DNF update Additional pointers and recipes on what to install the Oracle 11.2 on Fedora can is found on Tim Hall(link is external)' s website at:https://oracle-base.com/articles/11g/articles-11g(link is external)

Conclusions

BPF with its Bcc  frontend is new and  powerful tools  for dynamic Tracing for Linux. A discussion of the merits and shortfalls of BPF/BCC vs other existing solutions in the Linux dynamic tracing ecosystem are Beyond the scope of this post. What can find in this post is a few  example Scripts  that I have written for tracing Oracle using  bpf/bcc  and uprobes  and an additional script for integrating BPF/BCC with  pylatencymap , which provides  visualization as heat maps  of the histograms generated using BPF/BCC.

Although the BPF/BCC environment is currently under evolution, it appears already a very useful addition to the T Oolbox for troubleshooting and performance investigations of Linux workloads. A set of example Scripts/tools that come with the BCC repository is of great help for getting started both with Using BPF/BCC and with writing new scripts. BPF/BCC can only run on relatively new kernels (as I write this, I has tested the scripts discussed in this post On kernel version 4.6) and this is a obstacle for it adoption in many environments, at least .

Credits and Acknowledgements

Many of the original ideas and tools discussed here is inspired or directly derived from the awesome published work of Br Endan Gregg(link is external).

Many thanks also to the development teams of BPF and Bcc(link was external) for providing and supporting this new Powerful tools. In particular thanks to Brenden Blanco(link was external) for he work on Uprobes for BCC and for he support on I Ssue #478(link is external).

The work of investigating Oracle userspace with dynamic tracing tools have been do in collaboration with Frits Hoogland (link is external).

Linux bpf/bcc for Oracle tracing