November 14, 2012

Linux on System z presentation from the IBM System z Technical University Oct 2012

The Linux for System z specific presentations are now available online here.

Here are four presentation that I like to highlight:

October 5, 2012

Large pages in Linux on System z - howto enable in Linux

Since the z10 in 2008 the System z hardware offers large pages. The size is 1M and the main purpose is the same as on any other platform: reduce the memory footprint in the operating system for page tables and reduce the TLB misses in the hardware.
Linux on System z offers support for large pages. Right now they are not pageable, so only the required number should be allocated. There are two ways to enable it:
  1. Add hugepages=<npages> to the kernel parameter line, where npages is the number of 1M pages you want to allocate. So e.g. hugepages=1024 will allocate 1GB worth of memory as large pages.
  2. Dynamically: do a "echo <npages> > /proc/sys/vm/nr_hugepages" to receive npages worth of large pages. This only works if enough free memory is available that can be remapped as large pages. For 1GB of large pages "echo 1024 > /proc/sys/vm/nr_hugepages" will try to allocate them.
To check if the allocation worked, do a "cat /proc/sys/vm/nr_hugepages". Usage information can be as usual found in /proc/meminfo.

Note that if you are running in an environment, which does not support large pages, you can still enable them and get the memory savings as Linux will emulate large pages for the applications.

Finally if you are an application programmer and want to make use of large pages take a look at Documentation/vm/hugetlbpage.txt in the kernel source tree. There you find the supported interfaces for your kernel level.

September 12, 2012

zEC12 Linux certifications and tested platforms

IBM has announced a new mainframe the zEnterprise EC12. This systems supports all the Linux versions, that are currently supported by IBM, SUSE and Red Hat. Here is the detailed information:
Also for z/VM there are required PTFs that need to be installed. Also while you are installing, don't forget to check for additional PTFs marked as "Hiper".

August 7, 2012

40 years VM - happy birthday!

40 years ago IBM introduced VM/370 the first virtualization layer. My first really brief encounter was at the end of the 80ies, when I did some RPG programming work. But I learned the real value of it a decade later, when I joined IBM and we moved Linux to the mainframe. The tracing and debugging capabilities are really extremely useful for developing operating systems.

For further reading take a look at 

August 1, 2012

Oracle 11 installation problem on SLES 11 SP2

Before starting an installation of Oracle 11 with the grid infrastructure on SLES11 SP2 it's highly recommended to read the Oracle note with the title "OHASD fails to start on SuSE 11 SP2 on IBM: Linux on System z [ID 1476511.1]".
OHASD is the Oracle High Availability Service Daemon. So if this doesn't start on zLinux SLES 11 SP2 then consult the Oracle note above on

July 31, 2012

Getting mutrace to work on zLinux

Recently I got the question on how to trace mutex contention in libpthread on zLinux. There are several solutions to this
  1. Use SystemTap with the futexes.stp sample script
  2. Use Valgrind with the drd tool (see also this article) 
  3. Use mutrace as a lightweight tool
The last tool I've discovered in searching for solutions. However it hasn't been clear if it runs on zLinux or not. Usually it's just a ./configure and make to get a tool running but this one turned out to be a little bit more difficult.

I started of on a standard SLES11 SP2 with some of the development tools installed. So I downloaded the source from the mutrace-git and installed it in a directory. Then I called the ./ script. Sure enough it was failing:
+ aclocal -I m4 error: Autoconf version 2.68 or higher is required the top level
autom4te: /usr/bin/m4 failed with exit status: 63
aclocal: autom4te failed with exit status: 63

SLES11 SP2 hast autoconf 2.63, which isn't that ancient and SUSE had been patching and fixing it now in the second service pack. So I gave it a try and modified to accept a minimum level of 2.63. Next run:
checking for library containing bfd_init... no
configure: error: *** libbfd not found

This means that the system is missing the devel package of the binutils. After installing the binutil-devel package with
zypper install binutils-devel
the bootstrap script finished successfully. At the end I noted that it used a -O0 in the gcc options, which from a performance perspective is really bad on zLinux. So I changed that in the Makefile to a -O2.

So now only the compile had to work and sure enough it ended with a 
mutrace.c: In function setup:
mutrace.c:441: error: #pragma GCC diagnostic not allowed inside functions
mutrace.c:442: error: #pragma GCC diagnostic not allowed inside functions
mutrace.c:444: error: #pragma GCC diagnostic not allowed inside functions
make[1]: *** [libmutrace_la-mutrace.lo] Error 1

So this tool was using an advanced gcc feature that the gcc-4.3 from SUSE didn't have. Fortunately SUSE includes an updated version gcc-4.6 that can be installed along with the standard system compiler. The package name is gcc46 and instead of gcc you call gcc-4.6. After changing the Makefile once more the compile went smoothly.
Finally tried it on a small test program and it seems to work fine.
mutrace: Showing statistics for process a.out (PID: 12050).
mutrace: 1 mutexes used.

Mutex #0 (0x0x80003088) first referenced by:
        /root/mutrace-e23dc42/.libs/ [0x3fffd07c28e]
        ./a.out(functionCount1+0x20) [0x80000e34]
        /lib64/ [0x3fffd05436e]
        /lib64/ [0x3fffcfb417e]

mutrace: Showing 1 mutexes in order of (write) contention count:

 Mutex #   Locked  Changed    Cont. cont.Time[ms] tot.Time[ms] avg.Time[ms] Flags
       0       83       13        7         0.155        0.089        0.001 M-.--.
     ...      ...      ...      ...           ...          ...          ... ||||||
          Object:                                      M = Mutex, W = RWLock /||||
           State:                                  x = dead, ! = inconsistent /|||
             Use:                                  R = used in realtime thread /||
      Mutex Type:                   r = RECURSIVE, e = ERRORCHECK, a = ADAPTIVE /|
  Mutex Protocol:                                       i = INHERIT, p = PROTECT /

mutrace: Note that rwlocks are shown as two lines: write locks then read locks.

mutrace: Note that the flags column R is only valid in --track-rt mode!

mutrace: 1 condition variables used.

Condvar #0 (0x0x800030b0) first referenced by:
        /root/mutrace-e23dc42/.libs/ [0x3fffd07caea]
        ./a.out(functionCount1+0x32) [0x80000e46]
        /lib64/ [0x3fffd05436e]
        /lib64/ [0x3fffcfb417e]

mutrace: Showing 1 condition variables in order of wait contention count:

  Cond #    Waits  Signals    Cont. tot.Time[ms] cont.Time[ms] avg.Time[ms] Flags
       0        6       67        0        0.106         0.000        0.000     -.
     ...      ...      ...      ...          ...           ...          ...     ||
           State:                                     x = dead, ! = inconsistent /
             Use:                                     R = used in realtime thread

mutrace: Note that the flags column R is only valid in --track-rt mode!

mutrace: Total runtime is 0.319 ms.

mutrace: Results for SMP with 16 processors.

July 27, 2012

DB2 Connect high CPU utilization

DB2 Connect servers are usually a good target for consolidation. Recently we observed relatively high CPU utilization even though only a small workload was being used. The oprofile and strace output showed that the system was busy doing semget() calls that failed. So a resource was missing. In the end it turned out to be a known problem in DB2 which is fixed starting DB2 9.7 FP5 and can also be circumvented in older versions by issuing a "db2trc alloc" during startup.
This is a typical consolidation problem - on individual dedicated servers this usually is not even noticed - however after consolidation this will be visible. An additional positive effect of fixing the problem is an improved throughput at higher transaction rates.

July 4, 2012

New Whitepaper "Using the Linux cpuplugd Daemon to manage CPU and memory resources from z/VM Linux guests"

CPU and memory resources are normally shared in a virtualized environment. Therefore multiple guests are fighting for the same resources. Before the release of SLES11 SP2 and RHEL6.2 the automatic management of the number of virtual CPUs and the memory used in a virtual guest has been quite limited. Each system was requiring special attention and tuning of cpuplugd, the daemon that does the autonomic management in Linux on System z. Even then it had to be disabled for many systems.
The newer releases have a vastly improved daemon, that now allows for more detailed rules for adding and removing CPUs and memory to a guest. The drawback of more tuning knobs are more tuning knobs.
So this whitepaper tries to develop a recommended set of parameters to get the most benefit with the least effort. Furthermore results of measurements and experiments are shown together with the used parameters for advanced tuning.
Be aware that if you want cpuplugd to control memory and you run with more than one virtual CPU, you really want the fix for APAR VM65060 installed.
Also there has been a bug discovered in the base of SLES11 and RHEL 6 that's been fixed with maintweb kernel 3.0.42-0.7.3 for SLES 11 and the base RHEL 6.4 for Red Hat.

July 3, 2012

New Redpaper "Silent Installation Experiences with Oracle Database 11gR2 Real Application Clusters on Linux on System z"

IBM has published a new Redpaper covering the so called "silent mode" installation of Oracle. This installation method eliminates the GUI and user input during installation. Instead all parameters are specified in a file. Note that this applies to the latest Oracle only. This paper also covers the most current Oracle documents for installing on Linux on System z. Here is the content of this paper:
  • Oracle Database general information
  • Oracle environment
  • Oracle Grid Infrastructure silent mode installation
  • Oracle RAC silent mode installation
  • Upgrading with the latest patch set update
  • Optional methods to install and clone Oracle Database on Linux on z
    • Installing and cloning a single instance
    • Installing and cloning a two-node RAC database
    • Installing and cloning a new cluster
  • Cleaning up after a failed installation to perform a fresh installation
The standard installation of Oracle RAC is covered in "Installing Oracle 11gR2 RAC on Linux on System z" which I already recommended in an article here.

June 29, 2012

Fedora 17 for IBM System z 64bit official release

Dan HorĂ¡k from Red Hat announced the availability of the latest Fedora release for System z today. Congratulations to the team for making this available shortly after the release on the Intel platform!

The download is available from the Fedoraproject site and the respective mirrors and known issues and the new installation are covered in the wiki

June 25, 2012

Red Hat Enterprise Linux 6.3 released

Red Hat has announced the availability of RHEL 6.3. The kernel level is now kernel-2.6.32-279.el6, for the main bug fixes see the kernel update description.

As usual there are the release notes and the technical notes in two separate documents. The focus of this release has again been bug fixing. So a lot of fixes and minor enhancements especially in the s390utils have been delivered.

The IBM documentation for RHEL 6.3 for System z is on developerworks.

(update 8/15/2012)
If you are experiencing problems with accounting CPU time on RHEL 6.3 on System z, please upgrade to kernel 2.6.32-279.5.1.el6.s390x. This contains a very important fix for System z.

June 22, 2012

Linux distribution support for HX5 blades integrated in zBX

A number of people asked me about the supported Linux distributions for the Intel blade based zEnterprise BladeCenter Extensions. For this information look at

May 18, 2012

New article: Dynamic Program Analysis on Linux Using Valgrind

Did you have any memory problems with your applications recently? Wild accesses, improperly initialized variables, memory leaks, illegal frees etc sounds familiar? Then you should take a look at valgrind. It's now available for Linux on System z as well in SLES10 SP4, RHEL 6.1 (version 3.6) and SLES11 SP2 (version 3.7).
There is now an in depth article for usage on Linux on System z in the latest zJournal. Besides all the description it also has an example which shows you the usage of this powerful tool.

March 26, 2012

New Whitepaper: "Java Design and Coding for Virtualized Environments"

Ever wondered why your Java application is not running as smooth as it should in a virtualized environment? There are multiple reasons for this and so Steve Wehr gives some explanation for this in this new whitepaper. This isn't a complete answer and not everything is black and white but it should get you started. The topic covered in the paper are:
  • Why is Virtualization a Problem for Applications?
  • How to Waste CPU in your Java Application
  • How to Waste Memory in your Java Application
  • Best Practices for Java Applications in Virtualized Environments
  • Employ Strategies that Encourage Efficient Application Design for Virtualization
Of course there are other "performance bugs" as well, but from what I see in may daily work running applications in a virtualized environment with shared resources will make those bugs visible.

March 8, 2012

Whitepaper update: "Explore Decimal Floating Point support in Linux on System z"

Since a few years no System z offers hardware support for decimal floating point operations as defined by IEEE 754-2008. This allows for exact and very fast calculations in the decimal (and not the normal binary) system in the processor.

You can check if your system does support this by
cat /proc/cpuinfo | grep features
features        : esan3 zarch stfle msa ldisp eimm dfp edat etf3eh highgprs

If you see a "dfp" in included in the feature list, your system is enabled for decimal floating point.

The new instructions can be used from Java and C/C++. For Java use the java.math.BigDecimal class in a recent JVM. Other languages need compiler and library support and language extensions. ISO/IEC TR 24732 defines this for C and ISO/IEC TR 24733 for C++.

This updated whitepaper describes the details for what's needed to exploit this feature. It includes some performance results as well and all the necessary references

March 7, 2012

New Whitepaper: "WebSphere Application Server Horizontal Versus Vertical JVM Stacking Report"

This new whitepaper addresses the question if one should do a 1:1 consolidation or putting more than one server into a zLinux guest from a performance perspective. The sweet spot for the setup used here was at 10 JVMs per guest.
It also looks at the alternatives for sharing the application binaries between guests using Discontiguous Saved Segments (DCSS) and shared z/VM minidisks. Here it turned out that starting from 10 guests the DCSS setup was more efficient.

February 28, 2012

SLES 11 SP2 released

More than 21 months after SLES 11 SP1 today SUSE released the SP2 of their enterprise distribution. You can find more information on this release here:
As more information becomes available I'll update this post. Latest Update 5/18/2012.

February 23, 2012

Mainframe environments supported by Oracle

Recently there have been a lot of discussions with respect to what environments are supported for Oracle DB on the mainframe. On the Marist zLinux mailing list there is a good reply from a Oracle person. Here are a few more links for reference:

February 21, 2012

Red Hat Enterprise Linux 5.8 released

Red Hat has announced the availability of RHEL 5.8. You can find more information on this here:
From a mainframe perspective this has been a pure bug fix release. There is one important feature update though. Red Hat included an update to the additional gcc44 package enabling complete z196 support. As a dependency the binutils220 package will be installed as well. With those two packages you can start building binaries targeted for the z196 with the -march=z196 option or at least tune for it with options like -march=z9-109 -mtune=z196.

February 20, 2012

z196 GA2 feature base support in Linux

The required Linux distribution versions for the base support of new features like OSA Express4 are documented in the announcement letter (pdf). To find it scroll down to the Software requirements section.

February 6, 2012

Updated Virtualization Cookbook

The virtualization cookbook from Michael MacIsaac, Brad Hinson, Marian Gasparovic received a major updates to cover z/VM 6.2 (including singe system image), RHEL 6.2 and the upcoming SLES11 SP2. All is now in one book and it's even got a cloud touch with a chapter on xCAT. The code examples also received an update.

Linux and VIPA - white papers

In the past month a few questions regarding the use of OSPF and VIPA on Linux have been raised. This is especially important to clients combining z/OS and Linux and System z. So here are a two references to start with:
(updated 9/30/2013)

January 18, 2012

Red Hat Enterprise Linux 6.2 released

Red Hat has announced the availability of RHEL 6.2. New general features are described in this "what's new" document. The kernel level is now kernel-2.6.32-220.el6.

As usual there are the release notes and the technical notes in two separate documents. 

From the mainframe perspective the top focus has been on stability and reliability. Brad Hinson from Red Hat has done a nice summary of the changes focused on System z. Take a look at it!

My highlights for this release are:
  • Stability - many important bug fixes made it into this release. So if you are on RHEL6.0 or RHEL 6.1 consider upgrading!
  • Support for new crypto functions in the z196
  • Optimized math library for linear algebra functions, which is the ATLAS library optimized for System z
  • Access to raw ECKD data from Linux (read and write)
  • Lot's of updates in the s390utils package, most notably the enhancements for cpuplugd to autonomously manage CPU and memory resources in a virtualized environment.