December 20, 2013

Oprofile on zLinux - how to setup and use

(updated 1/8/2014)

Oprofile is a system wide profiler which is available in all major distributions for Linux on System z. Here is the setup and use for RHEL 6 and SLES 11.

The following example is for RHEL 6.5 (kernel 2.6.32-431.1.2.el6.s390x). For other kernel levels you need to adapt the package numbers. But basically it should work the same. The Red Hat description is here.

First step: install the required packages:
  • oprofile-0.9.7-1.el6.s390x
  • oprofile-jit-0.9.7-1.s390x (only needed for profiling Java code)
  • oprofile-gui-0.9.7-1.el6.s390x (only needed if you want the GUI)
  • kernel-debuginfo-2.6.32-431.1.2.el6.s390x.rpm
  • kernel-debuginfo-common-s390x-2.6.32-431.1.2.el6.s390x.rpm
Note that the kernel-debuginfo packages are only available on RHN. See this howto get it. You need to log into your Red Hat Customer portal for the full information. Also install from RHN any other debuginfo package of a distribution package you want to analyze. 

Second step: configure oprofile
opcontrol --setup --vmlinux=/usr/lib/debug/lib/modules/`uname -r`/vmlinux

Third step: measure workload
opcontrol --start
run your workload
opcontrol --stop
opcontrol --dump 

Last step: call opreport or opannotate with the options you want. For understanding options use the respective man pages. One commonly used option is:
opreport --symbols
Don't be surprised by an entry with symbol name vtime_stop_cpu. That's cpu  idle in RHEL 6.

For SLES 11 SP3 the setup is similar. SUSE has a good description on how to use in their Systems Analysis and Tuning Guide.

So in the first step you need to install oprofile-0.9.8-0.13.31.s390x.rpm from the SDK. Optionally  the kernel debuginfo package e.g. kernel-default-debuginfo-3.0.76-0.11.1.s390x.rpm as well as all the debuginfo versions of distribution packages you want to profile.

The vmlinux file for SLES is gzipped in /boot. If you have enough space there you can just gunzip it in place otherwise put it in /tmp as the SUSE guide suggests. Then in the second step you set up oprofile by
opcontrol --setup --vmlinux=/boot/vmlinux-`uname -r` 

Step 3 & 4 for SLES 11 are the same as above. 

If you want to analyze data on another system use oparchive. It will generate a directory with all required data that you can compress and take off the system. So e.g.
oparchive -p <path to Linux modules> -o /tmp/myoutputdir

You can also include Java and JITed code into the profiling by adding
-agentlib:jvmti_oprofile 
to your Java options. For SLES11 you need to add /usr/lib64/oprofile to your LD_LIBRARY_PATH. This especially valuable if you don't know yet where to search for a problem. If you have identified Java code as the problem then a specialized profiler is probably the better choice.


November 25, 2013

Installation of Linux on System z

Now and then I get the question on documentation for the base installation of Linux on System z. As this depends on the distribution, take a look at the documentation from the distribution partners first:
There is also a Live Virtual Class with a demonstration of some of the access methods. Charts and a replay are available for this.
The installation is also covered in several Redbooks, e.g. The Virtualization Cookbook for z/VM 6.3, RHEL 6.4 and SLES 11 SP3. 

November 22, 2013

Red Hat Enterprise Linux 6.5 released

Nine months after the last release Red Hat has announced the availability of RHEL 6.5. The kernel level is now kernel-2.6.32-431.el6, for the main bug fixes see the kernel update description.

As usual there are the release notes and the technical notes in two separate documents. For System z this has been a mainly a fix release. Only a few small enhancements have been added all described in the technical notes. Here are some of them:
The IBM documentation for RHEL 6.4 on developerworks still applies for this release.

November 18, 2013

Raw ECKD access from zLinux

Since a while the Linux DASD driver has been enhanced to access full ECKD tracks. Basically the following steps are needed:
  1. Take the device offline from other System z operating systems
  2. get the device online in zLinux 
  3. switch the device in raw ECKD mode:
    echo 1 > /sys/bus/ccw/devices/<device ID>/raw_track_access
  4. use a program that reads and writes whole tracks (64k) with direct IO like a dd with the respective flags to read or write
More details can be found in the chapter about the DASD driver (search for "raw_track_access") in the "Device Drivers, Features, and Commands" book for the respective distributions, e.g.
This book is really recommended as a reference.  

October 7, 2013

Red Hat Enterprise Linux 5.10 released

Red Hat has announced the availability of RHEL 5.10. You can find more information on this here:
This update has been the last release of RHEL 5 in the "Production 2"  phase. February of next year RHEL 5 will enter "Production 3". So from now on officially it's security and bug fixes only. For the exact details see the life cycle page from Red Hat.

From a System z perspective this is a pure bug fix release. No new features have been added.

September 24, 2013

Getting the Go programming language to work on zLinux

There is an increased interest in the new programming language Go. The standard download page doesn't have packages for s390x so I tried getting gccgo to work as described but with no success. Probably the current gccgo branch is broken for s390x.
So as a fallback I used the standard gcc 4.8.1. As the base I used a SLES 11 SP3. Here are the steps:
  1. Check out gcc 4.8.1 using svn
    svn checkout svn://gcc.gnu.org/svn/gcc/tags/gcc_4_8_1_release gcc481
  2. Create build directory (here build) and change to it
  3. Configure gcc with your favorite options, include go in it. I used
    ../gcc481/configure --prefix=/opt/gcc481 --enable-languages=c,c++,go --enable-shared --with-system-zlib --with-arch=z9-109 --enable-threads=posix --enable-__cxa_atexit --enable-checking --enable-lto
  4. The configure script is going to complain on various missing or outdated packages. So you probably need to try multiple times in a clean directory.
    Get the missing stuff from the SUSE DVDs and install them or build them from source. (I used gmp-4.2.3 and mpc2-0.8.2 from the SDK and compiled mpfr-3.1.2  from sources and installed in /usr/local)
  5. make
  6. make install
The result is an installation in /opt/gcc481. Before compiling any application you need to set the PATH and LD_LIBRARY_PATH to include the new environment:

export PATH=/opt/gcc481/bin:$PATH
export LD_LIBRARY_PATH=/opt/gcc481/lib64/:/opt/gcc481/lib/:$LD_LIBRARY_PATH

Finally you can enter the hellozworld.go program

package main
import "fmt"

func main() {
    fmt.Printf("Hello, z-world\n")
}

and compile and run it with

gccgo hellozworld.go -o hellozworld
./hellozworld

If you think this is an interesting language please approach Red Hat and SUSE and ask for it in the next releases!

September 12, 2013

New Whitepaper covering IBM Filenet P8 on zLinux

IBM Filenet P8 has been available for zLinux for quite some time. It's a really scalable document management, content lifecycle, and workflow platform. This is then used by Enterprise Content Management or Business Platform Management software. With that solution you can keep all the data secured on System z.
The new whitepaper "Linux on System z and IBM FileNet P8 5.1 Setup, Performance, and Scalability" (pdf version) covers in the performance and scalability on zLinux. It turns out that this is a solution well suited for the System z platform.

September 10, 2013

New Whitepaper "Implementing A Web Interface For The Linux Health Checker"

The Linux Health Checker is an IBM initiated open source project hosted on Sourceforge. It's a command line tool with an increasing number of plugins to find potential problems and wrong configurations before anything happens.
The new whitepaper describes how to automate the command line tool and collect all the reports from the whole z/VM in a singe web interface.

September 3, 2013

Updated Whitepaper: WebSphere Application Server - Idle Server Tuning

When running in a virtualized environment like z/VM it's beneficial if the hypervisor knows if a server is idle or not. Usually this is implemented by waiting a certain time before considering a server truly idle. The problem is that many of the applications and middleware products do housekeeping tasks way too often for this to be really effective. So any effort to lower this "noise" is good. The updated whitepaper "WebSphere Application Server - Idle Server Tuning" provides tuning suggestions for a Websphere environment to reduce this noise.
On top they also provide tuning recommendations to reduce the startup time. The team updated the paper to cover WAS v8 and v8.5.5 including the Liberty profile.

August 4, 2013

zBC12

On July 23rd IBM has announced the new business class model of the mainframe the zBC12. This is the successor to the z114 and is as you probably already guessed by the naming scheme build with the zEC12 technology.

From a Linux perspective the IBM tested platforms has been updated to include zBC12. Also Red Hat and SUSE updated their web sites to include the certifications:
Other useful links:
(updated 8/8/2013)

July 26, 2013

z/VM 6.3

After a long waiting time IBM released the latest version of z/VM. Now there is way more information available than just the little bit I described in my preview
The major items haven't changed from the preview - it's CPU scalability using Hiperdispatch and support for 1 TB per z/VM instance.

The announcement also has a statement of direction for z/VM 5.4.  Basically it says that the current System z generation (EC12 /BC12) will be the last one that's supported with z/VM 5.4. And support for z/VM 5.4 will end whenever the z9 hardware support ends. So it's time to plan for the upgrade to z/VM 6.3.

(updated 9/21/2013) 

July 24, 2013

Large Systems Performance Reference (LSPR) for zLinux

The Large Systems Performance Reference (LSPR) published by IBM has also a Linux workload included. There you can get a first impression of the relative performance about new System z processors.
More details can then be found in the z Processor Capacity Reference (zPCR).

July 16, 2013

Fedora 19 for IBM System z released

The new Fedora 19 release for System z / s390x happened today. Congratulations to the team!

The download is available from the Fedoraproject site and the respective mirrors and known issues  are covered in the wiki.

July 11, 2013

How to determine which kernel level corresponds to which service release for RHEL and SLES

Now and then I get the question - where to find the kernel level or version for a specific service release. And vice versa - I have kernel level x.y.z - what is the respective service level?
Both distributors SUSE as well as Red Hat offer this information on their web sites - even though not quite obvious to find. So here it is as a reference:

July 9, 2013

SLES 11 SP3 released

Today SUSE announced the release of Service Pack 3 for SLES 11. It's been more than 16 months since the last release, so quite some waiting time for new features. This time the kernel level stayed the same, so the upgrade from SLES11 SP2 to SLES11 SP3 should be easier than the one before.  You can find more information on this release here:
SUSE has released the first maintenance web kernel for SP3 (kernel-default-3.0.82-0.7.9.s390x.rpm). It contains more fixes than usual so as always when upgrading and starting, get the latest service installed. 

As more information becomes available I'll update this post. Latest Update 8/6/2013.

Cheat sheet for lock debugging in the Linux kernel

From time to time I'm getting a performance problems that requires identifying the lock in the Linux kernel that causes too much lock contention. The newer kernels are well equipped to help you find that lock. If you can do something that's another question.
The base documentation for that is in the kernel source under Documentation/locking/lockstat.txt. However due to the performance impact of all that tracing this usually is disabled in the distributions. For RHEL 6.4 there is a separate debug kernel that you need to install. Ensure that it's the default IPL/boot kernel or select in the IPL/boot menu.
SLES 11 is more difficult as this requires a  kernel rebuild with CONFIG_LOCK_STATS enabled. You need to contact SUSE service to get a kernel for your system.
If you have the system up with this enabled you should do the following:
  • echo 1 >/proc/sys/kernel/lock_stat
  • run your workload 
  • cat /proc/lock_stat > /tmp/lockreport.txt
  • echo 0 >/proc/sys/kernel/lock_stat
Usually you only need to take a look at the few top locks to find out what's going wrong.

June 21, 2013

zlib performance improvements

In my blog entry on RHEL 6.4 I've mentioned that there have been performance enhancements for zlib compression. However I never got around to actually measure this until today.

I've taken the zlib test program called mingzip.c from the Red Hat zlib 1.2.3 and linked it dynamically against libz. Then I created a 2 GB data file by taring up /usr/share in the Red Hat file system three times. So it has quite some compressible text in it. Finally I ran five rounds of compression for this file on each RHEL 6.3 and RHEL 6.4 with default compression and "-9" maximum compression.

The result for the Red Hat update is a +13% throughput increase for the maximum compression and still a +8% throughput increase for normal compression.Your mileage may vary of course.

The same test comparing a SLES 11.2 with the new SLES 11.3 (which has an upgrade to zlib 1.2.7) shows a +25% throughput increase for the maximum compression and still a +13% throughput increase for normal compression.
This is a relative comparison: the reason the numbers are higher on SLES is that SLES 11.2 is significantly slower in this test than RHEL 6.3. The latest releases (6.4 and 11.3) are showing again about the same throughput.

Everyone using applications that dynamically link against zlib gets an improvement automatically. For applications who either ship their own version of zlib or statically link against it, the vendor needs to pick up the patch and put it into the next version for this improvement.

June 18, 2013

Porting to Linux on System z

The recently released overview paper "Porting applications to Linux on IBM System z" prompted a few questions about porting in principal that I'll try to answer here in this blog entry.
(updated 7/15/2014)

June 3, 2013

New white paper: HyperPAV setup with z/VM and Red Hat Linux on zSeries

Parallel Access Volume (PAV)  allow you on System z to have more than one I/O outstanding per volume. However it's not so easy to set up and maintain, so this is why there is HiperPAV, which is quite easy to install and maintain. And it's supported by all in service Linux distributions now.

The white paper is gone from the IBM site - so the link is no longer working.
The new white paper / howto "HyperPAV setup with z/VM and Red Hat Linux on zSeries" describes the step by step setup for HyperPAV for z/VM and zLinux. So if you are using ECKD disks and have any I/O performance problems - make sure you've implemented this.

As this white paper is removed - here are a few pointers to get you started:

The presentation "z/VM PAV and HyperPAV Support" and the z/VM HyperPAV web page has a good overview from the z/VM side and the presentation "HyperPAV and Large Volume Support for Linux on System z" shows the Linux part (which is basically working out of the box). And there is of course the Virtualization Workbook, which covers HyperPAV as well.

The whitepaper has been removed from the IBM site - (updated 05/30/2015)


May 7, 2013

Linux and z/VM Live Virtual Classes 2013

There are regular Live Virtual Classes (LVC) for z/VM and Linux on System z. Watch the site to register for upcoming classes. There are also replays available for the older classes for z/VM, Linux on System z and z/VSE.
Here are the latest classes from 2013. The title links to the charts, the "replay" to the full replay.

May 2, 2013

New performance whitepaper on exploiting System z crypto hardware with WebSphere 8 for Linux on System z

System z offers hardware accelerations for all kinds of cryptographic operations. The new whitepaper "IBM WebSphere Application Server Version 8 for Linux on IBM System z – SSL Setup and Performance Study" shows how to set up WebSphere and Linux.
The performance numbers are quite impressive, especially for RSA 4096.

April 4, 2013

New article: Using Crypto Hardware with Java in Linux on System z

A new article on how to use the System z crypto hardware from Java inside Linux on System z has been published. The authors describe the available hardware for acceleration, the Linux software stack as it relates to Java. clear key and secure key handling.
However the most instructive part from my perspective is the Java example program and the "howto" description.

March 21, 2013

Java for Linux on System z - download and base documentation

There are several ways for getting the IBM JDK for Linux on System z
  • It's included with a lot of products like Websphere application server
  • It's included with the distributions delivered by SUSE and Red Hat
  • It can be downloaded from Developerworks
The documentation can be found here:
(updated 3/3/2015)

March 11, 2013

How to limit CPU usage of suspect runaway processes

Have you ever had the problem that a Linux guest under z/VM was using "too much" CPU and when you looked closer you identified a specific process in this guest? But you couldn't reach the application team, so just recycling this process isn't an option? And maybe it's still doing something reasonable, so a
kill -SIGSTOP [pid]
isn't an option.

The first option to solve the impact is to reduce the share from a z/VM perspective for this specific guest. So the usual SET SHARE ... But this will impact all processes in the guest, so if you have more than one application in a guest this isn't really an option.

The second option requires a tool called cpulimit. I've tried it on a SLES 11 SP2+ and here is what's needed to get it going. Download it from github and then build it on System z by calling make. Copy the binary to a default search location e.g. /usr/local/bin. Next find out the PID of the offending process e.g. by using top:

top - 14:58:30 up 11 min,  2 users,  load average: 12.38, 3.10, 1.19
Tasks: 109 total,   2 running, 107 sleeping,   0 stopped,   0 zombie
Cpu(s): 91.0%us,  0.1%sy,  0.0%ni,  8.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:     16124M total,     8645M used,     7479M free,        6M buffers
Swap:        0M total,        0M used,        0M free,      135M cached

  PID USER      PR  NI  VIRT  RES  SHR S   %CPU %MEM    TIME+  COMMAND
 3487 root      20   0 4724m  90m  11m S    909  0.6   4:41.35 java



Now limit this process to e.g. one CPU (I'm using 10 CPUs on this system) by

cpulimit --limit=100 --lazy --pid=3487

This tells the utility to limit to one CPU and exit if the process ends. I've had vmstat running while I entered this and the reduction is quite ok at the five second interval I've been using:

procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
12  0      0 7636040   6836 144752    0    0    11     1   88  118 26  0 74  0  0
33  0      0 7635816   6868 146000    0    0     0     2 2161 2634 89  0 11  0  0
34  0      0 7635816   6868 146120    0    0     0     0 2131 2576 89  0 11  0  0
36  0      0 7635816   6868 146236    0    0     0     0 2165 2640 89  0 11  0  0
 0  0      0 7635744   6868 146280    0    0     0     6  697 1421 15  0 84  0  0
 0  0      0 7635768   6868 146292    0    0     0     0  653 1454 11  0 89  0  0
 0  0      0 7635768   6868 146312    0    0     0     0  688 1371 11  0 89  0  0


Note that if you look at this in top, you still see quite some variance. Also don't expect it to be right to the last CPU cycles. But for the purposes here - reducing the CPU consumption down from 9 to 1 without entirely stopping the application - it does do the job.
There is one class of applications for which this doesn't work. Everything that needs an open terminal. As Rob correctly stated in his blog, this approach will disconnect and not reconnect the terminal again.

In newer distributions there is a third option called cgroups. Configured right, you should be able to move the offending PID into a limited group.

Everything should be tested on a test system first before tried on production images! 

February 22, 2013

Red Hat Enterprise Linux 6.4 released

Red Hat has announced the availability of RHEL 6.4. The kernel level is now kernel-2.6.32-358.el6, for the main bug fixes see the kernel update description.

As usual there are the release notes and the technical notes in two separate documents. For System z besides the usual preventive bug fixes also some interesting features have been made available:
The IBM documentation for RHEL 6.4 for System z is on developerworks.

February 12, 2013

Whitepapers for Live Guest Relocation and z/VM 6.2

(updated 9/2/2013)

The Platform Test team for System z has published two new white papers that should help readers to get a smooth start with Live Guest Relocation in z/VM 6.2.
Performance results are available here:
There are also two Redbooks published:
Even though the official term for mobility is "Live Guest Relocation" you might find also references to "Live Guest Mobility" or "Dynamic Relocation" or just "Mobility in z/VM".

February 11, 2013

Oracle 11 now certified for RHEL 6 on System z

After a long period of testing last week Oracle finally certified Oracle DB 11gR2 (11.2.0.3) running on RHEL 6.2 and later releases. For details see this Flash on TechDocs.With that all current distributions (RHEL5/6, SLES10/11) for Linux on System z are certified.
There also has been a webcast covering this hosted by the zSeries Oracle Special Interest Group.

February 5, 2013

z/VM 6.3 preview

Today IBM announced a preview of z/VM 6.3. The two top improvements are:
  • 1 TB memory 
  • HiperDispatch
At SHARE there have been two presentations with more technical details in it:

January 28, 2013

Fedora 18 for IBM System z released

The Fedora release 18 for System z is here. And this time Dan HorĂ¡k and the team did a great job as this happened only 8 days after the Intel release!

The download is available from the Fedoraproject site and the respective mirrors and known issues  are covered in the wiki.

January 10, 2013

Red Hat Enterprise Linux 5.9 released

Red Hat has announced the availability of RHEL 5.9. You can find more information on this here:
With this release in the life cycle of RHEL 5, the distribution is now in "Production 2". End of production is still a while out, but this means that from now on there will only be limited hardware enablement, no new software features and no new installation images. For the exact details see the life cycle page from Red Hat.

From a mainframe perspective there are two new features included in this release. VDSO will speed up certain system calls (e.g. gettimeofday) and HyperPAV will help FICON based I/O quite a bit. So please enable it, especially for Oracle databases!
An interesting enhancement in subscription manager now allows packages to be "locked" to a certain release. They won't be automatically updated with the next RHEL 5.10 if locked to e.g. 5.9.

January 8, 2013

New Whitepaper: "Oracle Database on Linux on System z - Disk I/O Connectivity Study"

For databases the IO performance is very important. This new whitepaper tries to develop guidelines for disk configuration and tuning hints for both ECKD and FCP devices. It also compares Oracle 10 and Oracle 11 for the transactional workload used. So if you needed a reason for an Oracle upgrade - there is another one.