Debugging Native heap OOM - JavaOne 2013

Matthew Kilner – IBM Java L3 Service – Core team lead
23rd September 2013

Debugging Native Heap OOM - Tools &
Techniques

© 2013 IBM Corporation

Important Disclaimers
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION
CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED.
ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED
ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR
INFRASTRUCTURE DIFFERENCES.
ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.
IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT
PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.
IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE
USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
- CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR
SUPPLIERS AND/OR LICENSORS

2


About me
 Matthew Kilner


 Work for IBM
– 13 years working on IBM Java
• Memory Management
• Class Sharing
• RAS
– Currently leading the Core customer support team.


 Contact info
– kilnerm@uk.ibm.com
– Twitter: @IBMJTC
– Youtube: IBM_JTC


 Visit the IBM booth #5112 and meet other IBM developers
at JavaOne 2013
3


What should you get from this talk?

 An understanding of what we mean by the Native Heap.

 A clear problem determination path for Native Heap OOM Errors:
– How to determine you have a native heap OOM.
– An outline process for determining what is causing it.


The Java process

 All applications run within the bounds of an operating system process
– Java is no exception
–
 The JVM is subject to the same restrictions as any other application, the most pertinent being:
• Addressing
• OS Memory Model


What do we mean by addressing restrictions?

 Every process has a finite address space which is dictated by the architecture it runs on.
–
 A 32bit architecture has an addressable range of:
– 2^32
–0x00000000 – 0xFFFFFFFF
–which is 4GB
–
 A 64bit architecture has an addressable range of:
–2^64
–0x0000000000000000 – 0xFFFFFFFFFFFFFFFF
–which is 16 EiB


What do we mean by Memory Model Restrictions?

 Not all addressable memory is available to a process.
 The operating system has its own requirements such as:
–The kernel
–The runtime support libraries
 Requirements vary by Operating System both in terms of:
– How much memory is needed, and
– Where that memory is located
 The addressable memory remaining is often referred to as “User Space”


A view by platform

 The chart shows default maximum user space available on common 32-bit platforms
9
8
7
6
5

GiB

4
3
2
1
0
Windows 32

Windows 32 /3GB

Linux 32 bit

Linux 32 bit
Hugemem

Operating System

zLinux 31

AIX 32

zOS

Kernel Space
User Space

What goes in the “User Space”
Kernel Space

 Java Heap
 Just In Time (JIT) Data
–Runtime data & executable code
 Virtual Machine Resources
–RAS engines & GC Infrastructure
 Native & JNI Allocations
 Resources to underpin Java Objects
–Classes and ClassLoaders
–Threads
–Direct java.nio.ByteBuffers
–Sockets

Java Heap

User Space

JIT Data
VM Resources

Native & JNI
Allocations
Java Libraries


The Native heap

 We define the native heap as:
–
–
Native Heap = “User Space” - Maximum Heap Size
–
 It is the total “User Space” not reserved for backing the Java Heap.




Not quite the whole story

 “User Space” availability is not our only consideration when looking at native memory shortage.

 Machines have to be able to back addressable memory with physical memory.

 The total physical memory available on a machine is

Physical RAM + Swap Space

 Problem symptoms vary based on which resource runs out.


Failure Symptoms

 The chart shows some of the symptoms you see when a particular resource is exhausted

Resource

Address Space

OutOfMemoryError
Symptoms

Physical RAM

OutOfMemoryError

Memory Pages

Console Messages
Crash/Other?

Physical + Swap
(Virtual Memory)

Unresponsive apps

Linux: OOM Killer
Win/Sol: alloc's fail

How do I know if I have a problem?

 Detecting a problem is easy if you hit one of the symptoms described previously.
– OutOfMemoryError's should be fatal to your application.
– Paging will cause obvious unresponsiveness or slowdown.

 Early detection is possible if you monitor the size of your process.
– Monitoring is also an important part of understanding any native memory issue.


Monitoring the size of your process

 Process sizes are reported in two ways across all platforms:
– Resident Size
– Virtual Size
 Each platform has its own methods for obtaining this information:
– Windows:
Performance Monitor
– Linux & z/OS:
ps
– AIX:
svmon
 The IBM Garbage Collection & Memory Visualizer (GCMV) tool provides scripts and instructions
in its help documentation for gathering the necessary data


Plotting the size of your process

 Analysis of the process size is best done visually.

 GCMV and Performance Monitor will plot the raw data for you.

 A persistent growth in the virtual size of your process may indicate an issue.




How do I identify a Native OOM from a javacore?

 When any OOME occurs the IBM JVM's default configuration will write a javacore file.

 This file provides several pointers to the fact you have an OOM related to the native heap:
– A header that includes information on which resource cannot be allocated
– Details of the current memory usage on the Java heap
– A record of recent Garbage Collection activity
– The stack of the thread encountering the problem





The javacore header

 At the top of each javacore is its header which tells you what event caused the file to be written.

 Under certain conditions additional information is written at the head of the javacore when the
OOME is triggered:
1TISIGINFO

Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal
-1073741830, errno 11" received


 On a Java Heap OOM you will see:
1TISIGINFO

Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Java heap space" received


The javacore heap usage summary

 Within the javacore you will find the MEMINFO section.

0SECTION
MEMINFO subcomponent dump routine
NULL
=================================
1STHEAPFREE Bytes of Heap Space Free: 3B3AE8
1STHEAPALLOC Bytes of Heap Space Allocated: 400000


 If you see a large value for bytes free then it is a good indicator that you are experiencing a native
heap OOME.




The javacore GC history

The javacore file also contains a snapshot of the most recent GC activity.
Where an OOM is due to a java heap allocation failing you will see this in the data:

1STGCHTYPE
3STHSTTYPE

GC History
14:12:41:476340000 GMT j9mm.101 - J9AllocateIndexableObject() returning NULL! 8000024 bytes requested for
object of class 00007FD4801D6E10 from memory space 'Flat'
id=00007FD480046EE8

If this entry is not present it is another good indicator you are experiencing a native heap
OOME.


The javacore current thread

The current thread within the javacore is the thread which triggered the OOME
– The top stack frame contains the interesting data
You can identify whether the frame is native:
3XMTHREADINFO
...............
3XMTHREADINFO3
4XESTACKTRACE
4XESTACKTRACE

"main" J9VMThread:0xB8D1C600, j9thread_t:0xB8D019E4, java/lang/Thread:0x98B01960, state:R, prio=5
Java callstack:
at java/lang/Thread.startImpl(Native Method)
at java/lang/Thread.start(Thread.java:891)

Or java:
3XMTHREADINFO
...............
3XMTHREADINFO3
4XESTACKTRACE

"main" J9VMThread:0x00007FD480043D00, j9thread_t:0x00007FD4800079B0,
Java callstack:
at StringOOM.main(StringOOM.java:11)

If it is native then this is another good indicator you are experiencing a native heap OOME.


How do I find out what is causing my native OOME?

 It can be tricky to attribute a root cause to a Native OOME
– Debug capabilities vary by platform.

 The fundamental approach is the same irrespective of platform:
• 1) Understand the rate of Native Memory Growth
• 2) Capture multiple snapshots of data over time.
• 3) Compare the snapshots and attribute growth to components.


Understanding the rate of memory growth

The rate of memory growth can be determined from the size of your process.

Calculate the delta in virtual size of your process between data snapshots.

Other data snapshots will identify different areas of native memory growth, understanding
the proportion each area contributes to the total growth is key to identifying a root cause.


What other data is needed?

Some data is common across platforms, other data is platform specific

Common data:
– Javacore files taken at regular intervals
– Core files taken at regular intervals (optional)

Platform specific data:
– Windows:
UMDH tracing, Debug Diag tracing, VMMAP tracing
– Linux:
No recommended tools
– AIX:
Debug malloc tracing


The javacore NATIVEMEMINFO section

From the J9 2.4 JVM the javacore file contains a NATIVEMEMINFO section.
NATIVEMEMINFO subcomponent dump routine
=======================================
JRE: 555,698,264 bytes / 1208 allocations
+--VM: 552,977,664 bytes / 856 allocations
| +--Classes: 1,949,664 bytes / 92 allocations
| +--Memory Manager (GC): 547,705,848 bytes / 146 allocations
| | +--Java Heap: 536,875,008 bytes / 1 allocation
| | +--Other: 10,830,840 bytes / 145 allocations
| +--Threads: 2,660,804 bytes / 104 allocations
| | +--Java Stack: 64,944 bytes / 9 allocations
| | +--Native Stack: 2,523,136 bytes / 11 allocations
| | +--Other: 72,724 bytes / 84 allocations
| +--Trace: 92,464 bytes / 208 allocations

| +--JVMTI: 17,328 bytes / 13 allocations
| +--JNI: 15,944 bytes / 32 allocations
| +--Port Library: 6,824 bytes / 56 allocations
| +--Other: 528,788 bytes / 205 allocations
+--JIT: 1,748,808 bytes / 82 allocations
| +--JIT Code Cache: 524,320 bytes / 1 allocation
| +--JIT Data Cache: 524,336 bytes / 1 allocation
| +--Other: 700,152 bytes / 80 allocations
+--Class Libraries: 971,792 bytes / 270 allocations
| +--Harmony Class Libraries: 1,024 bytes / 1 allocation
| +--VM Class Libraries: 970,768 bytes / 276 allocations
| | +--sun.misc.Unsafe: 69,688 bytes / 1 allocation
| | +--Other: 901,080 bytes / 275 allocations

Comparing the output from multiple javacores can identify areas of growth in the JVM
If an identified area is a significant portion of the total memory growth it is likely the cause
of the problem.

Javacores from earlier JDK versions

If your JDK is based on a JVM earlier than the J9 2.4 JVM the javacore file doe not have a
NATIVEMEMINFO section.

They do still contain valuable insight, although a little more work is required to obtain it.
– The “MEMINFO subcomponent dump routine” lists a summary of memory blocks the
JDK has allocated for various purposes
– The “Classes subcomponent dump routine” lists a summary of classloaders and loaded
classes.

By parsing and comparing this information across multiple javacore files you can
determine you have any signs of a memory area growth or classloader leak.

What core files offer

Binary core files provide the same view as the javacore but require processing with
external tools:
– Interactive Diagnostic Dump Explorer (IDDE).
– On earlier JDK versions they provide a more accurate summary of JDK memory
allocations the the javacore file.

They also provide a complete image of the process, which means:
– You can inspect the content of memory
– You can inspect free memory blocks (subject to platform)
– You can inspect allocated memory blocks (subject to platform)

These advantages are offset by the size of the files and additional overhead of processing
them.


Windows tooling

Windows offers three excellent options for understanding your native memory growth:
– UMDH
– DebugDiag
– VMMAP

Each has distinct usage characteristics:
– UMDH is command line driven.
– DebugDiag injects a tracking library into the process and parses core dumps.
– VMMAP launches the application you wish to track and is GUI based.


Linux tooling

Commercial tools are available but carry a license fee

Free tools also exist but carry a large performance overhead

We have custom built tooling that logs all calls to allocate and free memory
– Building your own is possible.


AIX tooling

 AIX provides a debug extension directly into the malloc subsystem
– MALLOCDEBUG
–
 Enables tracing in the allocation subroutines

 At termination of the process a report is generated detailing all allocations that were not freed
– Some additional parsing is needed


What the platform tooling tells you

While each platform has different tools, the end result from them is largely the same

The tools give you one or more stack traces that relate to memory allocations that have
not been freed.

You are looking for the stacks that demonstrate the same or similar rates of growth as the
total process size between snapshots of data.


What next?

The next step depends on the stack that has been identified as the root cause.
–
– If it is native code you own:
• Check to see you are releasing the memory your are allocating
–
–If it is native code relating to a java class:
• Check that you don't have an on heap leak of the related object type
• Check for known issues
• Contact the JDK vendor for assistance
–
–If it is third party native code:
• Check for any known issues
• Contact the vendor for assistance

In Summary

The process for diagnosis and root cause determination for a native OOME is as follows:
1) Understand the limitations of the platform
2) Monitor the size of the process to understand the rate of memory growth
3) Use a combination of JDK and platform diagnostics to determine the area or stack
driving the growth


I would like to know more

 Visit the IBM booth #5112

 BOF 4159 - The Most Useful Tools for Debugging on Windows
– Today: 9/23/13 (Monday) 7:30 PM - Hilton - Continental Ballroom 6

 “Thanks for the memory”
– http://www.ibm.com/developerworks/java/library/j-nativememory-linux/
– https://www.ibm.com/developerworks/java/library/j-nativememory-aix/
–



Questions?
–



IBM@JavaOne

http://ibm.co/JavaOne2013

35

IBM Confidential


Debugging Native heap OOM - JavaOne 2013

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Debugging Native heap OOM - JavaOne 2013 (20)

Recently uploaded (20)

Debugging Native heap OOM - JavaOne 2013