JavaOne2013: Build Your Own Runtime Monitoring for the IBM JDK with the Health Center API

© 2013 IBM Corporation
Chris Bailey – IBM Java Service Architect
26th
September 2013
Build your own Monitoring
With the Java Remote Monitoring API
Document number

© 2013 IBM Corporation
Important Disclaimers
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION
CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED.
ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED
ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR
INFRASTRUCTURE DIFFERENCES.
ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.
IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT
PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.
IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE
USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
- CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR
SUPPLIERS AND/OR LICENSORS
2

© 2013 IBM Corporation3
Introduction to the speaker
 13 years experience developing and deploying Java SDKs
 Recent work focus:
– Java applications in the cloud
– Java usability and quality
– Debugging tools and capabilities
– Requirements gathering
– Highly resilient and scalable deployments
 My contact information:
– baileyc@uk.ibm.com
– http://www.linkedin.com/in/chrisbaileyibm
– http://www.slideshare.net/cnbailey/

Goals of this talk
 Introduce the IBM tools for monitoring, debugging and diagnosing problems in the
deployment life cycle
 Understand the capabilities of Health Center
 Learn how to build your own tools using the Health Center data feed
 See how IBM is building on top of the Java tools platform

IBM Tools for Java

IBM Monitoring and Diagnostic Tools for Java
 A free suite of tools for understanding different aspects of Java applications
– Garbage Collection and Memory Visualizer (GCMV)
– Memory Analyzer (and IBM Extensions for Memory Analyzer)
– Interactive Diagnostic Data Explorer (IDDE)
– Health Center
 Provide visualization, analysis and recommendations
 Available in the IBM Support Assistant
– Provides the “toolbox” in which analysis tools reside
– Assist with opening PMRs and working with IBM Support
– Search feature to query IBM and non-IBM knowledge banks

Garbage Collection and Memory Visualizer
 Graphical Display of Data
– Allows graphing of all available data: pause
times, heap size etc
– Allows zoom, cropping and change of axes
value and units
– Allows comparison of multiple files
 Analysis and Recommendations
– Provides tuning recommendations based on
data and flags errors.
– Analysis can be limited using cropping.
– Values and units used in analysis can be
changed by changing axes values and units

Memory Analyzer
 Overview:
– Overview of the dump including Java heap
size and total number of objects.
– Provides links to continued analysis
 Path to GC Roots:
– Provides the reference chain that prevents
an object being garbage collected.
 Dominator Tree grouped by Class Loader:
– Lists the biggest objects using a “keep alive
tree”. Grouping by Class
– Loader limits the analysis to a single
application in a JEE environment
 Provides extension API
– Used to provide product based extensions

Interactive Diagnostic Data Explorer
 Multi-platform system dump analyzer
– AIX, Linux, i/OS, Windows, z/OS
 Provides dump portability
– Analyse dump on any platform
 Has knowledge of data structures in dump:
– Java, JVM, and some OS structures
 Provides collaborative analysis
– Store progress and notes with the dump
 Provides extension API
– Used to provide automated analysis

Health Center
 Live monitoring tool with very low overhead
– Pre-installed in IBM JDKs since Java 5
– Enable at startup: -Xhealthcenter
– Enable at runtime: java -jar healthcenter.jar ID=<pid>
 Understand how your application is behaving
– Monitor Class loading, File I/O, Environment settings, Garbage Collection, Allocations,
Method Profiling, Locking, Native memory use
 Diagnose potential problems, with recommendations
 Suitable for all Java applications

Health Center
 Environment reporting
– Detects invalid Java options
– Detects options which may hurt performance or
serviceability
– Useful for remote diagnosis of configuration-related
problems
 Class loading visualization
– Shows all loaded classes
– Shows load time
– Identifies shared classes
 File I/O
– Monitor application file open/close events as they occur
– Lists currently open files

Health Center
 Garbage Collection visualization
– Visualizes heap usage and gc pause times over time
– Identifies memory leaks
– Suggests command-line and tuning parameters
– Java Object allocation profiling by allocating code or type
 Native Memory
– Detect native memory leaks in application
– Determine if external forces are using more memory
– Memory counters showing which parts of the JVM are
using the most native memory
 Threads view
– List of current threads and states
– Number of threads over time
– See contended monitors

Health Center
 Method Profiling
– Always-on profiling shows application activity
– Identifies the hottest methods in an application
– Full call stacks to identify where methods are being called
from and what methods they call
– No byte code instrumentation, no recompiling
 Java Lock Profiling
– Always-on lock monitoring
– Quickly allows the usage of all locks to be profiled
– Helps to identify points of contention in the application
that are preventing scaling
 Live control of application
– Trigger dumps
– Enable verbosegc collection

Health Center Demo

Health Center: Overhead
 Measured using WebSphere App Server and the DayTrader benchmark with 50 clients
– Running WAS 8.5.5, IBM Java 7 SR5, AIX 7.1, POWER7
– Throughput determined by number of completed transactions on 4 saturated CPUs
94
95
96
97
98
99
100
101
102
103
100
Application Throughput
(percentage of baseline)
Baseline
Dormant (level=off)
Network Client
Headless client
(level=headless)
Headless client with ZIP
(level=headless)

94
95
96
97
98
99
100
101
102
103
100
101.76
Baseline
Dormant (level=off)
Network Client
Headless client
(level=headless)
(level=headless)

94
95
96
97
98
99
100
101
102
103
100
101.76
100.4
Baseline
Dormant (level=off)
Network Client
Headless client
(level=headless)
(level=headless)

94
95
96
97
98
99
100
101
102
103
100
101.76
100.4
99.01
Baseline
Dormant (level=off)
Network Client
Headless client
(level=headless)
(level=headless)

94
95
96
97
98
99
100
101
102
103
100
101.76
100.4
99.01
96.99
Baseline
Dormant (level=off)
Network Client
Headless client
(level=headless)
(level=headless)

Health Center
 Low overhead is achieved by surfacing existing data used by the JVM internally
– Performance overhead is therefore cost of storage or IO
 Garbage Collection Data:
– GC collects metrics to determine how to self tune
• Heap occupancy: expansion or contraction of the heap
• GC durations: expansion/contraction, concurrent kick-off
 Method Profiling:
– JIT compiler collects metrics to determine:
• When to compile: high use methods compiled first
• How to compile: optimize for most common flow through code
• When to recompile: assumptions are now incorrect

Java Monitoring API

Health Center and the Java Monitoring API
 Health Center obtains its data directly from the Java runtime and produces analysis and
recommendations shown in the Health Center client
 The Java runtime's data is now provided to any consumer via the Java Monitoring API
– Allows any tool to receive, analyze and interpret the data
– Allows those tools to additionally obtain the analysis and recommendations
 IBM Java Runtime provides a monitoring platform!

Using the API
 The API can be used in three modes:
– Remote: connecting over a socket connection (with security/authentication etc)
– Local: running inside the monitored application (using a local loop socket)
– Offline: loading a pre-saved binary “.hcd” file
 Performance cost to monitored application is the same as for Health Center
– Running in “local” mode adds has additional impact due to the the cost of parsing data

Using the API
 Three step process for obtaining data from the API:
 Obtain a Health Center object
– Using either a File or a Connection
 Obtain a data type object from the Health Center object
– getCpuData() / getGCData() / getThreadsData() / getIOData() / etc
 Request data points from the data type object:
– getCpuData().getProcessUse() / getCpuData.getSystemUse()

Obtain a Health Center Object: File
import com.ibm.java.diagnostics.healthcenter.api.ConnectionProperties;
import com.ibm.java.diagnostics.healthcenter.api.HealthCenter;
import com.ibm.java.diagnostics.healthcenter.api.HealthCenterException;
import com.ibm.java.diagnostics.healthcenter.api.factory.HealthCenterFactory;
public class GCGraphPrototype {
static HealthCenter hcAPI = null;
public static void main(String[] args) {
new GCGraphPrototype();
}
GCGraphPrototype() {
try {
hcAPI = HealthCenterFactory.connect(new File("C:healthcenter.hcd"));
} catch (HealthCenterException | FileNotFoundException e) {
e.printStackTrace();
}
testGCAPI();
}
...
Use factory to create Health Center Object
Connect to a File provider

Obtain a Health Center Object: Connection
}
ConnectionProperties props = new ConnectionProperties();
try {
hcAPI = HealthCenterFactory.connect(props, true);
} catch (HealthCenterException e) {
}
testGCAPI();
}
...
Describe your connection target: localhost:1972
Connect to your target

Obtain a Health Center Object: Connection
}
ConnectionProperties props = new ConnectionProperties("192.168.1.105", 35535));
try {
}
testGCAPI();
}
...
Describe a target of 192.168.1.105:35535

Obtain a Health Center Object: Connection Security
}
props.setJmxUsername("Username");
props.setJmxPassword("Password");
try {
}
testGCAPI();
}
...
Add JMX authentication

Obtain a Health Center Object: Connection Security
}
props.setSSLTruststoreLocation("Location");
props.setSSLTruststorePassword("Password");
try {
}
testGCAPI();
}
...
Add SSL encryption

testGCAPI():
private static void testGCAPI() {
GCGraph demo = new GCGraph();
demo.updateDataSet(hcAPI.getGCData().getGCPauseTimeData());
demo.pack();
demo.setVisible(true);
while (true) {
demo.updateDataSet(hcAPI.getGCData().getGCPauseTimeData());
demo.repaint();
try {
Thread.sleep(2 * 1000);
} catch (InterruptedException e) {
}
}
}
Get the GC Data provider
Get information on Pause Times

testGCAPI():
demo.updateDataSet(hcAPI.getGCData().getUsedHeapData());
demo.pack();
while (true) {
demo.updateDataSet(hcAPI.getGCData().getUsedHeapData());
demo.repaint();
try {
}
}
}
Get Java Heap usage information

testGCAPI():
demo.updateDataSet(hcAPI.getGCData().getHeapSizeData());
demo.pack();
while (true) {
demo.updateDataSet(hcAPI.getGCData().getHeapSizeData());
demo.repaint();
try {
}
}
}
Get Java Heap size information

testCPUAPI():
private static void testCPUAPI() {
demo.updateDataSet(hcAPI.getCpuData().getProcessUse());
demo.pack();
while (true) {
demo.updateDataSet(hcAPI.getCpuData().getProcessUse());
demo.repaint();
try {
}
}
}
Get the CPU Data provider
Get information on JVM CPU usage

testCPUAPI():
private static void testCPUAPI() {
demo.updateDataSet(hcAPI.getCpuData().getSystemUse());
demo.pack();
while (true) {
demo.updateDataSet(hcAPI.getCpuData().getSystemUse());
demo.repaint();
try {
}
}
}
Get information on overall CPU usage

GCGraph Class:
public class GCPauseGraph extends JFrame {
private XYSeriesCollection gcDataSet;
private static final long serialVersionUID = 1L;
public GCGraph() {
super();
gcDataSet = new XYSeriesCollection();
JFreeChart lineChartObject = ChartFactory.createXYLineChart(
“gc pauses”, "time (seconds)", "pause time (ms)",
getGCDataSet(), PlotOrientation.VERTICAL, true, true, false);
ChartPanel chartPanel = new ChartPanel(lineChartObject);
chartPanel.setPreferredSize(new java.awt.Dimension(500, 270));
setContentPane(chartPanel);
}
Extends Swing JFrame
Store data as X,Y values
Create a JFreeChart line plot
Add the JFreeChart to the JFrame

GCGraph Class:
public void updateDataSet(PauseData[] pD) {
final XYSeries series1 = new XYSeries("Pause Times");
if (pD != null) {
for (int i = 0; i < pD.length; i++) {
series1.add(pD[i].getTime() / 1000, pD[i].getPauseTime());
}
getGCDataSet().removeAllSeries();
getGCDataSet().addSeries(series1);
}
}
Create a JFree XYSeries
Populate with data
Remove old data
Add new data

Monitoring API Demo

Usage by IBM

Rational Application Developer 9.0
 In addition to supporting trace-based profiling, Rational Application Developer v9.0 Beta now
supports sample-based profiling for diagnosing execution time problems.
 Using the Monitoring API, introduced in Health Center v2.1
– Very low overhead compare with trace-based profiling.
– Available on all supported profiling launch types, including profiling on WAS servers.

Summary
 IBM is providing a very low overhead mechanism for accessing monitoring and profiling data
 Allows building of your own tools
 Same mechanism is being used by IBMs tools

http://ibm.co/JavaOne2013

References
 IBM Monitoring and Diagnostic Tools for Java™
– http://www.ibm.com/developerworks/java/jdk/tools/
 IBM Extensions for Memory Analyzer
– http://www.alphaworks.ibm.com/tech/iema
 Debugging WebSphere Application Server from Dumps (MA)
– http://www.ibm.com/developerworks/websphere/techjournal/1103_supauth/1103_supaut
h.html
 IBM Support Assistant (ISA)
– http://www.ibm.com/software/support/isa
 Email: javatool@uk.ibm.com

Copyright and Trademarks
© IBM Corporation 2013. All Rights Reserved.
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International
Business Machines Corp., and registered in many jurisdictions worldwide.
Other product and service names might be trademarks of IBM or other companies.
A current list of IBM trademarks is available on the Web – see the IBM “Copyright and
trademark information” page at URL: www.ibm.com/legal/copytrade.shtml

JavaOne2013: Build Your Own Runtime Monitoring for the IBM JDK with the Health Center API

More Related Content

What's hot (20)

Similar to JavaOne2013: Build Your Own Runtime Monitoring for the IBM JDK with the Health Center API (20)

More from Chris Bailey (20)

Recently uploaded (20)

JavaOne2013: Build Your Own Runtime Monitoring for the IBM JDK with the Health Center API