Oprofile on zLinux - how to setup and use

(updated 1/8/2014)

Oprofile is a system wide profiler which is available in all major distributions for Linux on System z. Here is the setup and use for RHEL 6 and SLES 11.

The following example is for RHEL 6.5 (kernel 2.6.32-431.1.2.el6.s390x). For other kernel levels you need to adapt the package numbers. But basically it should work the same. The Red Hat description is here.

First step: install the required packages:

• oprofile-0.9.7-1.el6.s390x
• oprofile-jit-0.9.7-1.s390x (only needed for profiling Java code)
• oprofile-gui-0.9.7-1.el6.s390x (only needed if you want the GUI)
• kernel-debuginfo-2.6.32-431.1.2.el6.s390x.rpm
• kernel-debuginfo-common-s390x-2.6.32-431.1.2.el6.s390x.rpm

Note that the kernel-debuginfo packages are only available on RHN. See this howto get it. You need to log into your Red Hat Customer portal for the full information. Also install from RHN any other debuginfo package of a distribution package you want to analyze. 

Second step: configure oprofile

opcontrol --setup --vmlinux=/usr/lib/debug/lib/modules/`uname -r`/vmlinux

Third step: measure workload

opcontrol --start

run your workload

opcontrol --stop

opcontrol --dump 

Last step: call opreport or opannotate with the options you want. For understanding options use the respective man pages. One commonly used option is:

opreport --symbols

Don't be surprised by an entry with symbol name vtime_stop_cpu. That's cpu  idle in RHEL 6.

For SLES 11 SP3 the setup is similar. SUSE has a good description on how to use in their Systems Analysis and Tuning Guide.

So in the first step you need to install oprofile-0.9.8-0.13.31.s390x.rpm from the SDK. Optionally  the kernel debuginfo package e.g. kernel-default-debuginfo-3.0.76-0.11.1.s390x.rpm as well as all the debuginfo versions of distribution packages you want to profile.

The vmlinux file for SLES is gzipped in /boot. If you have enough space there you can just gunzip it in place otherwise put it in /tmp as the SUSE guide suggests. Then in the second step you set up oprofile by

opcontrol --setup --vmlinux=/boot/vmlinux-`uname -r` 

Step 3 & 4 for SLES 11 are the same as above. 

If you want to analyze data on another system use oparchive. It will generate a directory with all required data that you can compress and take off the system. So e.g.

oparchive -p <path to Linux modules> -o /tmp/myoutputdir

You can also include Java and JITed code into the profiling by adding

-agentlib:jvmti_oprofile 

to your Java options. For SLES11 you need to add /usr/lib64/oprofile to your LD_LIBRARY_PATH. This especially valuable if you don't know yet where to search for a problem. If you have identified Java code as the problem then a specialized profiler is probably the better choice.