Oprofile on zLinux - how to setup and use

(updated 1/8/2014)

Oprofile is a system wide profiler which is available in all major distributions for Linux on System z. Here is the setup and use for RHEL 6 and SLES 11.

The following example is for RHEL 6.5 (kernel 2.6.32-431.1.2.el6.s390x). For other kernel levels you need to adapt the package numbers. But basically it should work the same. The Red Hat description is here.

First step: install the required packages:

• oprofile-0.9.7-1.el6.s390x
• oprofile-jit-0.9.7-1.s390x (only needed for profiling Java code)
• oprofile-gui-0.9.7-1.el6.s390x (only needed if you want the GUI)
• kernel-debuginfo-2.6.32-431.1.2.el6.s390x.rpm
• kernel-debuginfo-common-s390x-2.6.32-431.1.2.el6.s390x.rpm

Note that the kernel-debuginfo packages are only available on RHN. See this howto get it. You need to log into your Red Hat Customer portal for the full information. Also install from RHN any other debuginfo package of a distribution package you want to analyze. 

Second step: configure oprofile

opcontrol --setup --vmlinux=/usr/lib/debug/lib/modules/`uname -r`/vmlinux

Third step: measure workload

opcontrol --start

run your workload

opcontrol --stop

opcontrol --dump 

Last step: call opreport or opannotate with the options you want. For understanding options use the respective man pages. One commonly used option is:

opreport --symbols

Don't be surprised by an entry with symbol name vtime_stop_cpu. That's cpu  idle in RHEL 6.

For SLES 11 SP3 the setup is similar. SUSE has a good description on how to use in their Systems Analysis and Tuning Guide.

So in the first step you need to install oprofile-0.9.8-0.13.31.s390x.rpm from the SDK. Optionally  the kernel debuginfo package e.g. kernel-default-debuginfo-3.0.76-0.11.1.s390x.rpm as well as all the debuginfo versions of distribution packages you want to profile.

The vmlinux file for SLES is gzipped in /boot. If you have enough space there you can just gunzip it in place otherwise put it in /tmp as the SUSE guide suggests. Then in the second step you set up oprofile by

opcontrol --setup --vmlinux=/boot/vmlinux-`uname -r` 

Step 3 & 4 for SLES 11 are the same as above. 

If you want to analyze data on another system use oparchive. It will generate a directory with all required data that you can compress and take off the system. So e.g.

oparchive -p <path to Linux modules> -o /tmp/myoutputdir

You can also include Java and JITed code into the profiling by adding

-agentlib:jvmti_oprofile 

to your Java options. For SLES11 you need to add /usr/lib64/oprofile to your LD_LIBRARY_PATH. This especially valuable if you don't know yet where to search for a problem. If you have identified Java code as the problem then a specialized profiler is probably the better choice.

Installation of Linux on System z

Now and then I get the question on documentation for the base installation of Linux on System z. As this depends on the distribution, take a look at the documentation from the distribution partners first:

• RHEL 6 installation guide the System z specific topics are in Part III
• SLES 11 deployment guide - the System z specific topics are in Chapter 4

There is also a Live Virtual Class with a demonstration of some of the access methods. Charts and a replay are available for this.
The installation is also covered in several Redbooks, e.g. The Virtualization Cookbook for z/VM 6.3, RHEL 6.4 and SLES 11 SP3. 

Red Hat Enterprise Linux 6.5 released

Nine months after the last release Red Hat has announced the availability of RHEL 6.5. The kernel level is now kernel-2.6.32-431.el6, for the main bug fixes see the kernel update description.

As usual there are the release notes and the technical notes in two separate documents. For System z this has been a mainly a fix release. Only a few small enhancements have been added all described in the technical notes. Here are some of them:

• s390utils bug fix and enhancement update
• opencryptoki bug fix and enhancement update 
• adapter resiliency for crypto adapters
• fuzzy live dump
• support for large EAV volumes for installation
• Linux perf tool support for hardware performance counters
• VEPA mode support

The IBM documentation for RHEL 6.4 on developerworks still applies for this release.

Raw ECKD access from zLinux

Since a while the Linux DASD driver has been enhanced to access full ECKD tracks. Basically the following steps are needed:

1. Take the device offline from other System z operating systems
2. get the device online in zLinux 
3. switch the device in raw ECKD mode:
   echo 1 > /sys/bus/ccw/devices/<device ID>/raw_track_access
4. use a program that reads and writes whole tracks (64k) with direct IO like a dd with the respective flags to read or write

More details can be found in the chapter about the DASD driver (search for "raw_track_access") in the "Device Drivers, Features, and Commands" book for the respective distributions, e.g.

• Device Drivers, Features, and Commands on SUSE Linux Enterprise Server 11 SP3 - SC34-2595-03
• Device Drivers, Features, and Commands on Red Hat Enterprise Linux 6.4 - SC34-2597-04
• Device Drivers, Features, and Commands (kernel 3.11) - SC33-8411-22 

This book is really recommended as a reference.  

Red Hat Enterprise Linux 5.10 released

Red Hat has announced the availability of RHEL 5.10. You can find more information on this here:

• Announcement
• Release Notes
• Technical Notes
• Kernel update page including a list of bug fixes 
• kernel level: kernel-2.6.18-371.el5. 

This update has been the last release of RHEL 5 in the "Production 2"  phase. February of next year RHEL 5 will enter "Production 3". So from now on officially it's security and bug fixes only. For the exact details see the life cycle page from Red Hat.

From a System z perspective this is a pure bug fix release. No new features have been added.

Getting the Go programming language to work on zLinux

There is an increased interest in the new programming language Go. The standard download page doesn't have packages for s390x so I tried getting gccgo to work as described but with no success. Probably the current gccgo branch is broken for s390x.
So as a fallback I used the standard gcc 4.8.1. As the base I used a SLES 11 SP3. Here are the steps:

1. Check out gcc 4.8.1 using svn
   svn checkout svn://gcc.gnu.org/svn/gcc/tags/gcc_4_8_1_release gcc481
2. Create build directory (here build) and change to it
3. Configure gcc with your favorite options, include go in it. I used
   ../gcc481/configure --prefix=/opt/gcc481 --enable-languages=c,c++,go --enable-shared --with-system-zlib --with-arch=z9-109 --enable-threads=posix --enable-__cxa_atexit --enable-checking --enable-lto
4. The configure script is going to complain on various missing or outdated packages. So you probably need to try multiple times in a clean directory.
   Get the missing stuff from the SUSE DVDs and install them or build them from source. (I used gmp-4.2.3 and mpc2-0.8.2 from the SDK and compiled mpfr-3.1.2  from sources and installed in /usr/local)
5. make
6. make install

The result is an installation in /opt/gcc481. Before compiling any application you need to set the PATH and LD_LIBRARY_PATH to include the new environment:

export PATH=/opt/gcc481/bin:$PATH
export LD_LIBRARY_PATH=/opt/gcc481/lib64/:/opt/gcc481/lib/:$LD_LIBRARY_PATH

Finally you can enter the hellozworld.go program

package main
import "fmt"

func main() {
    fmt.Printf("Hello, z-world\n")
}

and compile and run it with

gccgo hellozworld.go -o hellozworld
./hellozworld

If you think this is an interesting language please approach Red Hat and SUSE and ask for it in the next releases!

New Whitepaper covering IBM Filenet P8 on zLinux

IBM Filenet P8 has been available for zLinux for quite some time. It's a really scalable document management, content lifecycle, and workflow platform. This is then used by Enterprise Content Management or Business Platform Management software. With that solution you can keep all the data secured on System z.
The new whitepaper "Linux on System z and IBM FileNet P8 5.1 Setup, Performance, and Scalability" (pdf version) covers in the performance and scalability on zLinux. It turns out that this is a solution well suited for the System z platform.

New Whitepaper "Implementing A Web Interface For The Linux Health Checker"

The Linux Health Checker is an IBM initiated open source project hosted on Sourceforge. It's a command line tool with an increasing number of plugins to find potential problems and wrong configurations before anything happens.
The new whitepaper describes how to automate the command line tool and collect all the reports from the whole z/VM in a singe web interface.

Updated Whitepaper: WebSphere Application Server - Idle Server Tuning

When running in a virtualized environment like z/VM it's beneficial if the hypervisor knows if a server is idle or not. Usually this is implemented by waiting a certain time before considering a server truly idle. The problem is that many of the applications and middleware products do housekeeping tasks way too often for this to be really effective. So any effort to lower this "noise" is good. The updated whitepaper "WebSphere Application Server - Idle Server Tuning" provides tuning suggestions for a Websphere environment to reduce this noise.
On top they also provide tuning recommendations to reduce the startup time. The team updated the paper to cover WAS v8 and v8.5.5 including the Liberty profile.

zBC12

On July 23rd IBM has announced the new business class model of the mainframe the zBC12. This is the successor to the z114 and is as you probably already guessed by the naming scheme build with the zEC12 technology.

From a Linux perspective the IBM tested platforms has been updated to include zBC12. Also Red Hat and SUSE updated their web sites to include the certifications:

• Red Hat certifications: RHEL5.8, RHEL 6.3
• SUSE certifications: SLES10 SP4, SLES11 SP2

Other useful links:

• IBM zEnterprise BC12 Technical Guide (Redbook) 
• zEC12 and zBC12 FAQ
• Red Hat promotion, SUSE promotion
• z/VM required service 

(updated 8/8/2013)

z/VM 6.3

After a long waiting time IBM released the latest version of z/VM. Now there is way more information available than just the little bit I described in my preview. 

• z/VM 6.3 general information (including all new features)
• announcement
• data sheet
• statement of direction
• special z/VM 6.3 resources page  
• "z/VM Version 6 Release 3: Making Room For Your Growing Business" in Enterprise System Journal 
• Updated z/VM performance report
• z/VM platform update Live Virtual Class and charts

The major items haven't changed from the preview - it's CPU scalability using Hiperdispatch and support for 1 TB per z/VM instance.

The announcement also has a statement of direction for z/VM 5.4.  Basically it says that the current System z generation (EC12 /BC12) will be the last one that's supported with z/VM 5.4. And support for z/VM 5.4 will end whenever the z9 hardware support ends. So it's time to plan for the upgrade to z/VM 6.3.

(updated 9/21/2013) 

Large Systems Performance Reference (LSPR) for zLinux

The Large Systems Performance Reference (LSPR) published by IBM has also a Linux workload included. There you can get a first impression of the relative performance about new System z processors.
More details can then be found in the z Processor Capacity Reference (zPCR).

Fedora 19 for IBM System z released

The new Fedora 19 release for System z / s390x happened today. Congratulations to the team!

The download is available from the Fedoraproject site and the respective mirrors and known issues  are covered in the wiki.

How to determine which kernel level corresponds to which service release for RHEL and SLES

Now and then I get the question - where to find the kernel level or version for a specific service release. And vice versa - I have kernel level x.y.z - what is the respective service level?
Both distributors SUSE as well as Red Hat offer this information on their web sites - even though not quite obvious to find. So here it is as a reference:

• Red Hat release table 
• SUSE release table including maintenance updates

SLES 11 SP3 released

Today SUSE announced the release of Service Pack 3 for SLES 11. It's been more than 16 months since the last release, so quite some waiting time for new features. This time the kernel level stayed the same, so the upgrade from SLES11 SP2 to SLES11 SP3 should be easier than the one before.  You can find more information on this release here:

• press release
• overview and list of new features for the mainframe
• release notes (s390x version) 
• release notes for High Availability Extensions (System z / s390x version) 
• release notes for SDK (System z / s390x version)
• kernel level: 3.0.76-0.9.1
• SUSE FAQ for SLES 11 SP3
• updated packages
• Here are some interesting features with links to the respective release notes entry:
• Transactional Execution Facility support
• libhugetlbfs support for transparent usage of large pages
• Optimized zlib compression library (see also my separate blog post) and upgrade to zlib 1.2.7
• Linux perf tool with hardware counters for performance analysis
• gcc 4.7 on SDK with zEC12 instructions and optimizations
• Crypto Express 4 (CEX4) support
• SHA-256 support in CCA token
• Flash Express support
• Fuzzy live dump
• kdump for Linux on System z
• and many more 
• Documentation
• SUSE page updated to SP3 level
• documentation as large zip file  (updated to SP3 level)
• IBM documentation on developerworks
• SUSE tuning guide updated to SP3 
• Oracle Database (10gR2 & 11gR2) on SUSE Linux Enterprise Server 11 SP3

SUSE has released the first maintenance web kernel for SP3 (kernel-default-3.0.82-0.7.9.s390x.rpm). It contains more fixes than usual so as always when upgrading and starting, get the latest service installed. 

As more information becomes available I'll update this post. Latest Update 8/6/2013.

Cheat sheet for lock debugging in the Linux kernel

From time to time I'm getting a performance problems that requires identifying the lock in the Linux kernel that causes too much lock contention. The newer kernels are well equipped to help you find that lock. If you can do something that's another question.
The base documentation for that is in the kernel source under Documentation/locking/lockstat.txt. However due to the performance impact of all that tracing this usually is disabled in the distributions. For RHEL 6.4 there is a separate debug kernel that you need to install. Ensure that it's the default IPL/boot kernel or select in the IPL/boot menu.
SLES 11 is more difficult as this requires a  kernel rebuild with CONFIG_LOCK_STATS enabled. You need to contact SUSE service to get a kernel for your system.
If you have the system up with this enabled you should do the following:

• echo 1 >/proc/sys/kernel/lock_stat
• run your workload 
• cat /proc/lock_stat > /tmp/lockreport.txt
• echo 0 >/proc/sys/kernel/lock_stat

Usually you only need to take a look at the few top locks to find out what's going wrong.

zlib performance improvements

In my blog entry on RHEL 6.4 I've mentioned that there have been performance enhancements for zlib compression. However I never got around to actually measure this until today.

I've taken the zlib test program called mingzip.c from the Red Hat zlib 1.2.3 and linked it dynamically against libz. Then I created a 2 GB data file by taring up /usr/share in the Red Hat file system three times. So it has quite some compressible text in it. Finally I ran five rounds of compression for this file on each RHEL 6.3 and RHEL 6.4 with default compression and "-9" maximum compression.

The result for the Red Hat update is a +13% throughput increase for the maximum compression and still a +8% throughput increase for normal compression.Your mileage may vary of course.

The same test comparing a SLES 11.2 with the new SLES 11.3 (which has an upgrade to zlib 1.2.7) shows a +25% throughput increase for the maximum compression and still a +13% throughput increase for normal compression.
This is a relative comparison: the reason the numbers are higher on SLES is that SLES 11.2 is significantly slower in this test than RHEL 6.3. The latest releases (6.4 and 11.3) are showing again about the same throughput.

Everyone using applications that dynamically link against zlib gets an improvement automatically. For applications who either ship their own version of zlib or statically link against it, the vendor needs to pick up the patch and put it into the next version for this improvement.

Porting to Linux on System z

The recently released overview paper "Porting applications to Linux on IBM System z" prompted a few questions about porting in principal that I'll try to answer here in this blog entry.

• The developerWorks article "Porting applications to Linux for System z" provides more technical details, especially on the differences between 31bit and 64bit and of course the endianess problem. So before you start porting your C/C++ program take a look at this. The migration toolkit can be found here and here.
• Open Source packages not yet integrated in a distribution are usually pretty easy. Most of the time a simple ./configure, make, make install sequence does the job. You probably need to install a few development libraries before. Sometimes small changes to the make file are required to recognize s390x as a big-endian 64 bit architecture. Usually you can take a look at the ppc64 implementation.  
• Also for Open Source projects there is the possibility to get access to a System z using the  Community Development System for Linux on System z (CDSL). 
• Java workloads usually run just out of the box. 
• Don't forget to download and install the latest service pack though. 
• Due to the 31bit addressing the heap of the 31bit JVM on System z can't be as large as the heap on a 32bit JVM. So for e.g. a 3 GB heap you need to switch to the 64 bit JVM on zLinux. If you  then use the -Xcompressedrefs option you can keep the additional memory reasonable.
• For ISVs there is at IBM Partnerworld a special roadmap called "Porting your UNIX or Linux on x86 solution to Linux for IBM System z mainframe server platforms". 
• Also hosted on Partnerworld is the "IBM Systems Application Advantage for Linux (Chiphopper)". Which is an offering from IBM to help porting ISV applications.
• IBM als runs Porting Centers around the world for professional help. Also available is the IBM Migration Factory.
• For Open Source projects there is the Community Development System for Linux on System z (CDSL). Register there to get access to a System z. 

(updated 7/15/2014)

New white paper: HyperPAV setup with z/VM and Red Hat Linux on zSeries

Parallel Access Volume (PAV)  allow you on System z to have more than one I/O outstanding per volume. However it's not so easy to set up and maintain, so this is why there is HiperPAV, which is quite easy to install and maintain. And it's supported by all in service Linux distributions now.

The white paper is gone from the IBM site - so the link is no longer working.
The new white paper / howto "HyperPAV setup with z/VM and Red Hat Linux on zSeries" describes the step by step setup for HyperPAV for z/VM and zLinux. So if you are using ECKD disks and have any I/O performance problems - make sure you've implemented this.

As this white paper is removed - here are a few pointers to get you started:

The presentation "z/VM PAV and HyperPAV Support" and the z/VM HyperPAV web page has a good overview from the z/VM side and the presentation "HyperPAV and Large Volume Support for Linux on System z" shows the Linux part (which is basically working out of the box). And there is of course the Virtualization Workbook, which covers HyperPAV as well.

The whitepaper has been removed from the IBM site - (updated 05/30/2015)

Linux and z/VM Live Virtual Classes 2013

There are regular Live Virtual Classes (LVC) for z/VM and Linux on System z. Watch the site to register for upcoming classes. There are also replays available for the older classes for z/VM, Linux on System z and z/VSE.
Here are the latest classes from 2013. The title links to the charts, the "replay" to the full replay.

• How to surprise by being a Linux Performance "know-it-all" - replay
  This class covers some of the more advanced performance tooling on Linux on System z
• Linux on System z Disk I/O Performance - Part 1- replay
• zEC12 Performance Update - replay
• zFCP Performance Analysis - replay
  This class covers the details of the ziomon tool suite on zLinux to analyze FCP performance problems. This tooling is the way to go for in depth analysis of problems but it is not intended for everyday monitoring. There the syssstat tool suite is certainly the better approach.
• Live Demo - z/VM Live Guest Relocation - replay
• z/VM 6.2 Performance Update - replay
• Running Linux-HA on a IBM System z - replay

New performance whitepaper on exploiting System z crypto hardware with WebSphere 8 for Linux on System z

System z offers hardware accelerations for all kinds of cryptographic operations. The new whitepaper "IBM WebSphere Application Server Version 8 for Linux on IBM System z – SSL Setup and Performance Study" shows how to set up WebSphere and Linux.
The performance numbers are quite impressive, especially for RSA 4096.

New article: Using Crypto Hardware with Java in Linux on System z

A new article on how to use the System z crypto hardware from Java inside Linux on System z has been published. The authors describe the available hardware for acceleration, the Linux software stack as it relates to Java. clear key and secure key handling.
However the most instructive part from my perspective is the Java example program and the "howto" description.

Java for Linux on System z - download and base documentation

There are several ways for getting the IBM JDK for Linux on System z

• It's included with a lot of products like Websphere application server
• It's included with the distributions delivered by SUSE and Red Hat
• The new download site on developer.ibm.com
• Old download page on Developerworks

The documentation can be found here:

• Linux User guides for Java 1.4, Java 5 and Java 6
• Diagnosis guides for Java 1.4, Java 5 and Java 6
• Java 7:  All in one Java Information Center. For the diagnosis guide choose "IBM User Guide for Java V7 on Linux" --> "Troubleshooting and support"
• Java 8: Java Information Center. For the diagnosis guide choose "Linux User Guide for IBM SDK" --> "Troubleshooting and support"
• JDK end of support and lifcycle information

(updated 10/23/2017)

How to limit CPU usage of suspect runaway processes

Have you ever had the problem that a Linux guest under z/VM was using "too much" CPU and when you looked closer you identified a specific process in this guest? But you couldn't reach the application team, so just recycling this process isn't an option? And maybe it's still doing something reasonable, so a
kill -SIGSTOP [pid]
isn't an option.

The first option to solve the impact is to reduce the share from a z/VM perspective for this specific guest. So the usual SET SHARE ... But this will impact all processes in the guest, so if you have more than one application in a guest this isn't really an option.

The second option requires a tool called cpulimit. I've tried it on a SLES 11 SP2+ and here is what's needed to get it going. Download it from github and then build it on System z by calling make. Copy the binary to a default search location e.g. /usr/local/bin. Next find out the PID of the offending process e.g. by using top:

top - 14:58:30 up 11 min,  2 users,  load average: 12.38, 3.10, 1.19
Tasks: 109 total,   2 running, 107 sleeping,   0 stopped,   0 zombie
Cpu(s): 91.0%us,  0.1%sy,  0.0%ni,  8.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:     16124M total,     8645M used,     7479M free,        6M buffers
Swap:        0M total,        0M used,        0M free,      135M cached

  PID USER      PR  NI  VIRT  RES  SHR S   %CPU %MEM    TIME+  COMMAND
 3487 root      20   0 4724m  90m  11m S    909  0.6   4:41.35 java


Now limit this process to e.g. one CPU (I'm using 10 CPUs on this system) by

cpulimit --limit=100 --lazy --pid=3487

This tells the utility to limit to one CPU and exit if the process ends. I've had vmstat running while I entered this and the reduction is quite ok at the five second interval I've been using:

procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
12  0      0 7636040   6836 144752    0    0    11     1   88  118 26  0 74  0  0
33  0      0 7635816   6868 146000    0    0     0     2 2161 2634 89  0 11  0  0
34  0      0 7635816   6868 146120    0    0     0     0 2131 2576 89  0 11  0  0
36  0      0 7635816   6868 146236    0    0     0     0 2165 2640 89  0 11  0  0
 0  0      0 7635744   6868 146280    0    0     0     6  697 1421 15  0 84  0  0
 0  0      0 7635768   6868 146292    0    0     0     0  653 1454 11  0 89  0  0
 0  0      0 7635768   6868 146312    0    0     0     0  688 1371 11  0 89  0  0

Note that if you look at this in top, you still see quite some variance. Also don't expect it to be right to the last CPU cycles. But for the purposes here - reducing the CPU consumption down from 9 to 1 without entirely stopping the application - it does do the job.
There is one class of applications for which this doesn't work. Everything that needs an open terminal. As Rob correctly stated in his blog, this approach will disconnect and not reconnect the terminal again.

In newer distributions there is a third option called cgroups. Configured right, you should be able to move the offending PID into a limited group.

Everything should be tested on a test system first before tried on production images! 

High Availibility solutions for Websphere MQ and Linux on System z

The standard approach for achieving high availability with Websphere MQ is to use a High Availability Manager and a shared DASD. The following whitepapers describe this for the available options:

• High Availability Solution for WebSphere Message Queue in a single instance queue manager configuration using Tivoli SAfMP and a shared DASD device
• High Availability Solution for WebSphere Message Queue in a Single Instance Queue Manager Configuration using HAE and a Shared DASD Device
• High Availability Solution for WebSphere Message Queue in a Single Instance Queue Manager Configuration Using Linux HA and a Shared DASD Device

Red Hat Enterprise Linux 6.4 released

Red Hat has announced the availability of RHEL 6.4. The kernel level is now kernel-2.6.32-358.el6, for the main bug fixes see the kernel update description.

As usual there are the release notes and the technical notes in two separate documents. For System z besides the usual preventive bug fixes also some interesting features have been made available:

• FCP end-to-end data checking 
• s390utils enhancements
• Storage Class Memory aka Flash Express support
• Support for the Transactional-Execution Facility in zEC12
• Crypto Express 4S (CEX4S) adapter card support in z90crypt 
• libica enhancements  
• improved compression performance for zlib. Details can be found here. 
• ...

The IBM documentation for RHEL 6.4 for System z is on developerworks.

Whitepapers for Live Guest Relocation and z/VM 6.2

(updated 9/2/2013)

The Platform Test team for System z has published two new white papers that should help readers to get a smooth start with Live Guest Relocation in z/VM 6.2.

• Early Experiences with z/VM 6.2 and Live Guest Relocation: moving from CSE to SSI:  This paper describes the installation differences to a classical z/VM installation as experienced by the test team. It's not replacing the product documentation but rather augmenting it.
• z/VM Single System Image Cluster Live Guest Relocation with HiperSockets: Exploitation via Dynamic Routing: Live Guest Relocation and HiperSockets look like a contradiction at the first glance. If you move a z/VM  guest off the CEC - what do you do with this network interface? This paper offers a clever approach to this problem, while keeping the solution as flexible as possible. 
• z/VM Migration: Migrating the User Directory and RACF Environment: This document describes an automated migration to a new SSI environment using REXX scripts.

Performance results are available here:

• z/VM 6.2 Live Guest Relocation with Linux Middleware: This paper looks at a Java application and the impact and performance for moving it in various scenarios. It also gives recommendations on how to best set up z/VM for Live Guest Relocation
• z/VM 6.2 Live Guest Relocation with Linux Middleware - Various Workloads: This is the follow on paper to the one above with additional workloads and more in depth analysis. Besides looking deeper at Java it also uses a OLTP database workload and a file system workload for comparison and analysis.
• The z/VM performance update (starting at page 16) at SHARE contains quite some performance results for LGR.

There are also two Redbooks published:

• An Introduction to z/VM Single System Image (SSI) and Live Guest Relocation (LGR): This really is an introduction to the new z/VM functionality
• Using z/VM v 6.2 Single System Image (SSI) and Live Guest Relocation (LGR): This book is a little bit more advanced with concrete examples of moving e.g. an Websphere and a DB2 guest under load.

Even though the official term for mobility is "Live Guest Relocation" you might find also references to "Live Guest Mobility" or "Dynamic Relocation" or just "Mobility in z/VM".

Oracle 11 now certified for RHEL 6 on System z

After a long period of testing last week Oracle finally certified Oracle DB 11gR2 (11.2.0.3) running on RHEL 6.2 and later releases. For details see this Flash on TechDocs.With that all current distributions (RHEL5/6, SLES10/11) for Linux on System z are certified.
There also has been a webcast covering this hosted by the zSeries Oracle Special Interest Group.

z/VM 6.3 preview

Today IBM announced a preview of z/VM 6.3. The two top improvements are:

• 1 TB memory 
• HiperDispatch

At SHARE there have been two presentations with more technical details in it:

• z/VM performance update (starting at page 57) 
• z/VM Platform update (starting at page 44)

Fedora 18 for IBM System z released

The Fedora release 18 for System z is here. And this time Dan Horák and the team did a great job as this happened only 8 days after the Intel release!

The download is available from the Fedoraproject site and the respective mirrors and known issues  are covered in the wiki.

Red Hat Enterprise Linux 5.9 released

Red Hat has announced the availability of RHEL 5.9. You can find more information on this here:

• Announcement
• Release Notes
• Technical Notes
• Kernel update page including a list of bug fixes 
• kernel level: kernel-2.6.18-348.el5. 

With this release in the life cycle of RHEL 5, the distribution is now in "Production 2". End of production is still a while out, but this means that from now on there will only be limited hardware enablement, no new software features and no new installation images. For the exact details see the life cycle page from Red Hat.

From a mainframe perspective there are two new features included in this release. VDSO will speed up certain system calls (e.g. gettimeofday) and HyperPAV will help FICON based I/O quite a bit. So please enable it, especially for Oracle databases!
An interesting enhancement in subscription manager now allows packages to be "locked" to a certain release. They won't be automatically updated with the next RHEL 5.10 if locked to e.g. 5.9.

New Whitepaper: "Oracle Database on Linux on System z - Disk I/O Connectivity Study"

For databases the IO performance is very important. This new whitepaper tries to develop guidelines for disk configuration and tuning hints for both ECKD and FCP devices. It also compares Oracle 10 and Oracle 11 for the transactional workload used. So if you needed a reason for an Oracle upgrade - there is another one.