Recently I got the question on how to trace mutex contention in libpthread on zLinux. There are several solutions to this
- Use SystemTap with the futexes.stp sample script
- Use Valgrind with the drd tool (see also this article)
- Use mutrace as a lightweight tool
The last tool I've discovered in searching for solutions. However it hasn't been clear if it runs on zLinux or not. Usually it's just a
./configure and
make to get a tool running but this one turned out to be a little bit more difficult.
I started of on a standard SLES11 SP2 with some of the development tools installed. So I downloaded the source from the
mutrace-git and installed it in a directory. Then I called t
he ./bootstrap.sh script. Sure enough it was failing:
...
+ aclocal -I m4configure.ac:21: error: Autoconf version 2.68 or higher is requiredconfigure.ac:21: the top levelautom4te: /usr/bin/m4 failed with exit status: 63aclocal: autom4te failed with exit status: 63
SLES11 SP2 hast autoconf 2.63, which isn't that ancient and SUSE had been patching and fixing it now in the second service pack. So I gave it a try and modified
configure.ac to accept a minimum level of 2.63. Next run:
...
checking for library containing bfd_init... no
configure: error: *** libbfd not found
This means that the system is missing the devel package of the binutils. After installing the binutil-devel package with
zypper install binutils-develthe bootstrap script finished successfully. At the end I noted that it used a
-O0 in the gcc options, which from a performance perspective is really bad on zLinux. So I changed that in the Makefile to a
-O2.
So now only the compile had to work and sure enough it ended with a
mutrace.c: In function setup:
mutrace.c:441: error: #pragma GCC diagnostic not allowed inside functions
mutrace.c:442: error: #pragma GCC diagnostic not allowed inside functions
mutrace.c:444: error: #pragma GCC diagnostic not allowed inside functions
make[1]: *** [libmutrace_la-mutrace.lo] Error 1
So this tool was using an advanced gcc feature that the gcc-4.3 from SUSE didn't have. Fortunately SUSE includes an updated version gcc-4.6 that can be installed along with the standard system compiler. The package name is gcc46 and instead of
gcc you call
gcc-4.6. After changing the Makefile once more the compile went smoothly.
Finally tried it on a small test program and it seems to work fine.
mutrace: Showing statistics for process a.out (PID: 12050).mutrace: 1 mutexes used.Mutex #0 (0x0x80003088) first referenced by: /root/mutrace-e23dc42/.libs/libmutrace.so(pthread_mutex_lock+0x9e) [0x3fffd07c28e] ./a.out(functionCount1+0x20) [0x80000e34] /lib64/libpthread.so.0(+0x836e) [0x3fffd05436e] /lib64/libc.so.6(+0xef17e) [0x3fffcfb417e]mutrace: Showing 1 mutexes in order of (write) contention count: Mutex # Locked Changed Cont. cont.Time[ms] tot.Time[ms] avg.Time[ms] Flags 0 83 13 7 0.155 0.089 0.001 M-.--. ... ... ... ... ... ... ... |||||| /||||| Object: M = Mutex, W = RWLock /|||| State: x = dead, ! = inconsistent /||| Use: R = used in realtime thread /|| Mutex Type: r = RECURSIVE, e = ERRORCHECK, a = ADAPTIVE /| Mutex Protocol: i = INHERIT, p = PROTECT / RWLock Kind: r = PREFER_READER, w = PREFER_WRITER, W = PREFER_WRITER_NONRECmutrace: Note that rwlocks are shown as two lines: write locks then read locks.mutrace: Note that the flags column R is only valid in --track-rt mode!mutrace: 1 condition variables used.Condvar #0 (0x0x800030b0) first referenced by: /root/mutrace-e23dc42/.libs/libmutrace.so(pthread_cond_wait+0x7a) [0x3fffd07caea] ./a.out(functionCount1+0x32) [0x80000e46] /lib64/libpthread.so.0(+0x836e) [0x3fffd05436e] /lib64/libc.so.6(+0xef17e) [0x3fffcfb417e]mutrace: Showing 1 condition variables in order of wait contention count: Cond # Waits Signals Cont. tot.Time[ms] cont.Time[ms] avg.Time[ms] Flags 0 6 67 0 0.106 0.000 0.000 -. ... ... ... ... ... ... ... || /| State: x = dead, ! = inconsistent / Use: R = used in realtime threadmutrace: Note that the flags column R is only valid in --track-rt mode!mutrace: Total runtime is 0.319 ms.mutrace: Results for SMP with 16 processors.