The Ubuntu Mainframe Cookbook

If you liked the Virtualization Cookbooks for Red Hat and SUSE and you are looking at Ubuntu right now then maybe you are interested in Mike MacIsaac's latest book: The Ubuntu Mainframe Cookbook.

Example for using Apache Spark on the mainframe

This YouTube video from Paul DiMarzio shows an interesting analytic example from the Hack Spark contest. Thanks Paul for creating this!

Middleware / application testing - operating system, hypervisor and hardware

After the IBM statement of directions to support KVM on System z I've been asked many times for a criteria if a certain ISV software or customer solution needs to be retested or not. As in performance the answer is: it depends. Basically there are two / four different types of applications / products to consider

1. No retesting needed - certified and tested towards the operating system:
   Applications that use a standard API only, like Java or standard library functions that don't need and exploit the visibility of the underlying (virtualized) hardware.
2. Retesting recommended - certified and tested towards the operating system:
   Applications that do interact with virtualized hardware to a certain extend e.g. accessing raw devices.
3. Retesting and some coding needed - certified and tested towards the operating system and the virtualization layer:
   Applications that actively interact with the hypervisor, e.g. by vmcp commands under z/VM. They then needs an equivalent implementation of that kind of functionality for every type of virtualization. Other examles are the use of zLinux hypfs (hypervisor file system) or using the hypervisor details surfaced in /proc/sysinfo.
4. Retesting and additional code for the hypervisor needed - certified and tested towards the operating system and the virtualization layer:
   Applications that need hooks and special interfaces into the hypervisor e.g. for monitoring or systems management. Every new hypervisor will require new code as e.g. the reported performance metrics and systems management interfaces are different

For Linux on the mainframe today there is only z/VM and LPAR. And z/VM is really virtualizing the hardware in a 1:1 fashion as good as possible (with a lot of firmware and hardware support to keep the overhead to minimal levels).  For such a hardware virtualization the category #2 is nearly not existent. If there is no active interaction with hypervisor, no retesting is needed.
KVM has the concept of new virtual devices with new device names, so some products from #1 are going to slip into category #2.

(updated 4/15/2015)

Updated Whitepaper: WebSphere Application Server - Idle Server Tuning

When running in a virtualized environment like z/VM it's beneficial if the hypervisor knows if a server is idle or not. Usually this is implemented by waiting a certain time before considering a server truly idle. The problem is that many of the applications and middleware products do housekeeping tasks way too often for this to be really effective. So any effort to lower this "noise" is good. The updated whitepaper "WebSphere Application Server - Idle Server Tuning" provides tuning suggestions for a Websphere environment to reduce this noise.
On top they also provide tuning recommendations to reduce the startup time. The team updated the paper to cover WAS v8 and v8.5.5 including the Liberty profile.

z/VM 6.3 preview

Today IBM announced a preview of z/VM 6.3. The two top improvements are:

• 1 TB memory 
• HiperDispatch

At SHARE there have been two presentations with more technical details in it:

• z/VM performance update (starting at page 57) 
• z/VM Platform update (starting at page 44)

40 years VM - happy birthday!

40 years ago IBM introduced VM/370 the first virtualization layer. My first really brief encounter was at the end of the 80ies, when I did some RPG programming work. But I learned the real value of it a decade later, when I joined IBM and we moved Linux to the mainframe. The tracing and debugging capabilities are really extremely useful for developing operating systems.

For further reading take a look at 

• Official IBM site
• 40 Reflections for 40 Years of z/VM
• z/VM – A Brief Review of Its 40 Year History

DB2 Connect high CPU utilization

DB2 Connect servers are usually a good target for consolidation. Recently we observed relatively high CPU utilization even though only a small workload was being used. The oprofile and strace output showed that the system was busy doing semget() calls that failed. So a resource was missing. In the end it turned out to be a known problem in DB2 which is fixed starting DB2 9.7 FP5 and can also be circumvented in older versions by issuing a "db2trc alloc" during startup.
This is a typical consolidation problem - on individual dedicated servers this usually is not even noticed - however after consolidation this will be visible. An additional positive effect of fixing the problem is an improved throughput at higher transaction rates.

New Whitepaper: "Java Design and Coding for Virtualized Environments"

Ever wondered why your Java application is not running as smooth as it should in a virtualized environment? There are multiple reasons for this and so Steve Wehr gives some explanation for this in this new whitepaper. This isn't a complete answer and not everything is black and white but it should get you started. The topic covered in the paper are:

• Why is Virtualization a Problem for Applications?
• How to Waste CPU in your Java Application
• How to Waste Memory in your Java Application
• Best Practices for Java Applications in Virtualized Environments
• Employ Strategies that Encourage Efficient Application Design for Virtualization

Of course there are other "performance bugs" as well, but from what I see in may daily work running applications in a virtualized environment with shared resources will make those bugs visible.

Updated Virtualization Cookbook

The virtualization cookbook from Michael MacIsaac, Brad Hinson, Marian Gasparovic received a major updates to cover z/VM 6.2 (including singe system image), RHEL 6.2 and the upcoming SLES11 SP2. All is now in one book and it's even got a cloud touch with a chapter on xCAT. The code examples also received an update.

Why doesn't the Linux iowait show up as busy time in the virtualization layer?

When a process in Linux has issued an IO and is waiting for a response there are basically two different possibilities. The good case is that there is other work pending and the CPU can continue to work. If there is no work left to do the CPU changes it's state to idle and the time until either new work comes in or the IO is delivered is accounted as %iowait in tools like vmstat, iostat or sar. So you can view "iowait" as a shade of "idle".

From the perspective of the hipervisor such an idle CPU can be put to good use e.g. by dispatching it to another guest that's in need of CPU. This is why from a virtualization layer perspective guest CPUs in iowait are usually accounted as idle.
For problem determination it would help to add an iowait measure to the hipervisor as well as it would help detecting problems that are created by cloning inefficient guests, e.g. 100
servers all doing sync IO where async IO would have been possible.