December 20, 2013

Oprofile on zLinux - how to setup and use

(updated 1/8/2014)

Oprofile is a system wide profiler which is available in all major distributions for Linux on System z. Here is the setup and use for RHEL 6 and SLES 11.

The following example is for RHEL 6.5 (kernel 2.6.32-431.1.2.el6.s390x). For other kernel levels you need to adapt the package numbers. But basically it should work the same. The Red Hat description is here.

First step: install the required packages:
  • oprofile-0.9.7-1.el6.s390x
  • oprofile-jit-0.9.7-1.s390x (only needed for profiling Java code)
  • oprofile-gui-0.9.7-1.el6.s390x (only needed if you want the GUI)
  • kernel-debuginfo-2.6.32-431.1.2.el6.s390x.rpm
  • kernel-debuginfo-common-s390x-2.6.32-431.1.2.el6.s390x.rpm
Note that the kernel-debuginfo packages are only available on RHN. See this howto get it. You need to log into your Red Hat Customer portal for the full information. Also install from RHN any other debuginfo package of a distribution package you want to analyze. 

Second step: configure oprofile
opcontrol --setup --vmlinux=/usr/lib/debug/lib/modules/`uname -r`/vmlinux

Third step: measure workload
opcontrol --start
run your workload
opcontrol --stop
opcontrol --dump 

Last step: call opreport or opannotate with the options you want. For understanding options use the respective man pages. One commonly used option is:
opreport --symbols
Don't be surprised by an entry with symbol name vtime_stop_cpu. That's cpu  idle in RHEL 6.

For SLES 11 SP3 the setup is similar. SUSE has a good description on how to use in their Systems Analysis and Tuning Guide.

So in the first step you need to install oprofile-0.9.8-0.13.31.s390x.rpm from the SDK. Optionally  the kernel debuginfo package e.g. kernel-default-debuginfo-3.0.76-0.11.1.s390x.rpm as well as all the debuginfo versions of distribution packages you want to profile.

The vmlinux file for SLES is gzipped in /boot. If you have enough space there you can just gunzip it in place otherwise put it in /tmp as the SUSE guide suggests. Then in the second step you set up oprofile by
opcontrol --setup --vmlinux=/boot/vmlinux-`uname -r` 

Step 3 & 4 for SLES 11 are the same as above. 

If you want to analyze data on another system use oparchive. It will generate a directory with all required data that you can compress and take off the system. So e.g.
oparchive -p <path to Linux modules> -o /tmp/myoutputdir

You can also include Java and JITed code into the profiling by adding
-agentlib:jvmti_oprofile 
to your Java options. For SLES11 you need to add /usr/lib64/oprofile to your LD_LIBRARY_PATH. This especially valuable if you don't know yet where to search for a problem. If you have identified Java code as the problem then a specialized profiler is probably the better choice.