IBM AIX: Java process size monitoring

This article will provide you with a quick reference guide on how to calculate the Java process size memory footprint for Java VM processes running on IBM AIX 5.3+ OS.

This is a complementary post to my original article on this subject: how to monitor the Java native memory on AIX. I highly recommend this read to any individual involved in production support or development of Java applications deployed on AIX.

Why is this knowledge important?

From my perspective, basic knowledge on how the OS is managing the memory allocation of your JVM processes is very important. We often overlook this monitoring aspect and only focus on the Java heap itself.

From my experience, most Java memory related problems are observed from the Java heap itself such as garbage collection problems, leaks etc. However, I’m confident that you will face situations in the future involving native memory problems or OS memory challenges. Proper knowledge of your OS and virtual memory management is crucial for proper root causes analysis, recommendations and solutions.

AIX memory vs. pages

As you may have seen from my earlier post, the AIX Virtual Memory Manager (VMM) is responsible to manage memory requests from the system and its applications.

The actual physical memory is converted and partitioned in units called pages; allocated either in physical RAM or stored on disk until it is needed. Each page can have a size of 4 KB (small page), 64 KB (medium page) or 16 MB (large page). Typically for a 64-bit Java process you will see a mix of all of the above.

What about the topas command?

The typical reflex when supporting applications on AIX is to run the topas command, similar to Solaris top. Find below an example of output from AIX 5.3:

As you can see, the topas command is not very helpful to get a clear view on the memory utilization since it is not providing the breakdown view that we need for our analysis. It is still useful to get a rough idea of the paging space utilization which can give you a quick idea of your top "paging space" consumer processes. Same can be achieved via the ps aux command.

AIX OS command to the rescue: svmon

The AIX svmon command is by far my preferred command to deep dive into the Java process memory utilization. This is a very powerful command, similar to Solaris pmap. It allows you to monitor the current memory “pages” allocation along with each segment e.g. Java Heap vs. native heap segments. Analyzing the svmon output will allow you to calculate the memory footprint for each page type (4 KB, 64 KB, and 16 MB).

Now find below a real example which will allow you to understand how the calculation is done:

# 64-bit JVM with -Xms2048m & -Xmx2048m (2 GB Java Heap)

# Command: svmon –P <Java PID>

As you can see, the total footprint of our Java process size was found at 2.2 GB which is aligned with current Java heap settings. You should be able to easily perform the same memory footprint analysis from your AIX environment

I hope this article has helped you to understand how to calculate the Java process size on AIX OS. Please feel free to post any comment or question.

Read More

JRockit jrcmd tutorial

This article will provide you an overview and tutorial on how you can perform an initial analysis and problem isolation of a JRockit Java Heap problem using the jrcmd tool. A deeper analysis and tutorial using JRockit Mission Control and Heap Dump analysis (JRockit version R28+ only) will be covered in future articles.

For a quick overview of the JRockit Java Heap Space, please consult the article below:

JRockit Java Heap Space

JRCMD tool overview

jrcmd is a free tool that is available out-of-the-box in the JRockit binaries. It allows you generate and collect crucial data from your runtime JRockit VM such as:

-        Java process memory space breakdown (Java Heap vs Native memory spaces)

-        Java Heap diagnostic (histogram)

-        Java loaded classes

-        On-demand JRockit Heap Dump generation (version R28+ only)

-        Thread Dump generation

-        More…

For this article, we created a simple Java program leaking internally. We will use this program to demonstrate how you can leverage jrcmd to perform your initial analysis.

Sample Java memory leak program

This simple Java program is simply adding String data to a static HashMap and slowly leaking to the point of the JVM running out of Java Heap memory. This program will allow you to visualize a slowly growing Java Heap leak via JRockit jrcmd. Please note that a Java Heap size of 128 MB (-Xms128m –Xmx128m) was used for this example.

/**

 * JavaHeapLeakSimulator

 * @author Pierre-Hugues Charbonneau

 * http://javaeesupportpatterns.blogspot.com

 */

public class JavaHeapLeakSimulator {

        private final static int NB_ITERATIONS = 500000000;

        // ~1 KB data footprint

        private final static String LEAKING_DATA_PREFIX = "datadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadata";

        // Map used to stored our leaking String instances

        private static Map<String, String> leakingMap;

        static {

               leakingMap = new HashMap<String, String>();

        }

        /**

         * @param args

         */

        public static void main(String[] args) {

               System.out.println("Java Heap Leak Simulator 1.0");

               System.out.println("Author: Pierre-Hugues Charbonneau");

               System.out.println("http://javaeesupportpatterns.blogspot.com/");

               try {

                       for (int i = 0; i < NB_ITERATIONS; i++) {

                              String data = LEAKING_DATA_PREFIX + i;

                             

                              // Add data to our leaking Map data structure...

                              leakingMap.put(data, data);

                             

                              // Slowdown the Java program so we can monitor the leak before the OutOfMemoryError condition

                              Thread.sleep(1);

                       }

               } catch (Throwable any) {

                       if (any instanceof java.lang.OutOfMemoryError) {

                              System.out.println("OutOfMemoryError triggered! "

                                             + any.getMessage() + " [" + any + "]");

                       } else {

                              System.out.println("Unexpected Exception! " + any.getMessage()

                                             + " [" + any + "]");

                       }

               }

               System.out.println("JavaHeapLeakSimulator done!");

        }

}

JRCMD - initial execution

JRCMD can be executed from the local server hosting the JVM that you want to monitor or remotely via JRockit Mission Control. The executable is located within the JRockit JDK that you are using:

<JRockit_JDK_HOME>/bin/jrcmd

The default jrcmd execution will return the list of active JRockit Java process Id that you can monitor:

C:\Apps\Weblogic1035\jrockit_160_24_D1.1.2-4\bin>jrcmd

5360 org.ph.javaee.tool.oom.JavaHeapLeakSimulator

5952

6852 jrockit.tools.jrcmd.JrCmd

JRCMD - Java Heap monitoring

The next step is to start monitoring the Java Heap memory usage and histogram. A Java Heap histogram is a snapshot of the biggest pools of Java Class instances. This allow you to pinpoint the leaking data type. Ple

You can either chose between print_object_summary (quick summary) or heap_diagnostics (complete breakdown).

C:\Apps\Weblogic1035\jrockit_160_24_D1.1.2-4\bin>jrcmd 5360 heap_diagnostics

Invoked from diagnosticcommand

======== BEGIN OF HEAPDIAGNOSTIC =========================

Total memory in system: 8465022976 bytes

Available physical memory in system: 5279170560 bytes

-Xmx (maximal heap size) is 134217728 bytes

Heapsize: 134217728 bytes

Free heap-memory: 123592704 bytes

--------- Detailed Heap Statistics: ---------

90.9% 3948k     5468  +3948k [C

 3.0% 128k     5490   +128k java/lang/String

 2.1% 92k     3941    +92k java/util/HashMap$Entry

 1.2% 50k      461    +50k java/lang/Class

 0.8% 35k       21    +35k [Ljava/util/HashMap$Entry;

 0.6% 24k        7    +24k [B

 0.3% 15k      305    +15k [Ljava/lang/Object;

 0.3% 14k      260    +14k java/net/URL

 0.2% 6k      213     +6k java/util/LinkedHashMap$Entry

 0.1% 4k      211     +4k java/io/ExpiringCache$Entry

 0.1% 2k        4     +2k [Ljrockit/vm/FCECache$FCE;

 0.0% 1k       50     +1k [Ljava/lang/String;

 0.0% 1k       10     +1k java/lang/Thread

 0.0% 1k       61     +1k java/util/Hashtable$Entry

 0.0% 1k        7     +1k [I

 0.0% 0k       19     +0k java/util/HashMap

 0.0% 0k       19     +0k java/lang/ref/WeakReference

 0.0% 0k        7     +0k [Ljava/util/Hashtable$Entry;

 0.0% 0k       19     +0k java/util/Locale

 0.0% 0k       11     +0k java/lang/ref/SoftReference

 0.0% 0k        1     +0k [S

…………………………………………………

- The first column correponds to the Class object type contribution to the Java Heap footprint in %
- The second column correponds to the Class object type memory footprint in K
- The third column correponds to the # of Class instances of a particular type
- The fourth column correponds to the delta - / + memory footprint of a particular type

As you can see from the above snapshot, the biggest data type is [C (char in our case) & java.lang.String. In order to see which data types are leaking, you will need to generate several snapshots. The frequency will depend of the leaking rate. In our example, find below another snapshot taken after 5 minutes:

# After 5 minutes

--------- Detailed Heap Statistics: ---------

93.9% 26169k    28746 +12032k [C

 2.4% 674k    28768   +295k java/lang/String

 2.3% 637k    27219   +295k java/util/HashMap$Entry

 0.9% 259k       21   +128k [Ljava/util/HashMap$Entry;

 0.2% 50k      462     +0k java/lang/Class

 0.1% 24k        7     +0k [B

# After 5 more minutes

--------- Detailed Heap Statistics: ---------

94.5% 46978k    50534 +20809k [C

 2.4% 1184k    50556   +510k java/lang/String

 2.3% 1148k    49007   +510k java/util/HashMap$Entry

 0.5% 259k       21     +0k [Ljava/util/HashMap$Entry;

 0.1% 50k      462     +0k java/lang/Class

The third & fourth column are showing a constant increase. As you can see, the leaking data in our case are [C, java.lang.String and java.util.HashMap$Entry which all increased from ~4 MB to 28 MB, 50 MB and growing…

It is easy to pinpoint the leaking data type(s) with this approach but what about the source (root cause) of the leaking data type(s)? This is where jrcmd is no longer useful. Deeper memory leak analysis will require you to use either JRockit Mission Control or Heap Dump analysis (JRockit R28+ only).

A final point, before you can conclude on a true Java Heap leak, please ensure that jrcmd snapshots are taken after at least one Full GC in between the captures (what you are interested in is OldGen leak e.g. Java objects surviving major GC collections). 

JRCMD Thread Dump generation

Thread Dump analysis is crucial for stuck Thread related problems but can also be useful to troubleshoot certain types of Java Heap problem. For example, it can pinpoint the source of a sudden Java Heap increase by exposing the culprit Thread(s) allocating a large amount of memory on the Java Heap in a short amount of time. Thread Dump can be generated using jrcmd print_threads option.

** Thread Dump captured from our sample Java program after removing the Thread.sleep() and increasing the Java Heap capacity **

C:\Apps\Weblogic1035\jrockit_160_24_D1.1.2-4\bin>jrcmd 5808 print_threads

5808:

===== FULL THREAD DUMP ===============

Mon Apr 09 09:08:08 2012

Oracle JRockit(R) R28.1.3-11-141760-1.6.0_24-20110301-1429-windows-ia32

"Main Thread" id=1 idx=0x4 tid=6076 prio=5 alive, native_blocked

    at jrockit/vm/Allocator.getNewTla(II)V(Native Method)

    at jrockit/vm/Allocator.allocObjectOrArray(Allocator.java:354)[optimized]

    at java/util/Arrays.copyOfRange(Arrays.java:3209)[inlined]

    at java/lang/String.<init>(String.java:215)[inlined]

    at java/lang/StringBuilder.toString(StringBuilder.java:430)[optimized]

    at org/ph/javaee/tool/oom/JavaHeapLeakSimulator.main(JavaHeapLeakSimulator.java:38)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

……………………………………….

We can see that our sample Java program is creating a lot of java.lang.String objects from the “Main Thread” executing our JavaHeapLeakSimulator program.

Conclusion

I hope this article has helped you understand you can leverage the JRockit jrcmd tool for quick Java Heap analysis. I’m looking forward for your comments and questions.

Future articles will include a deeper JRockit Java Heap and Heap Dump analysis tutorial.

Read More

Plumbr: Java memory leak detector

This article is to inform you that I will be performing a review and trial of the Plumbr Java memory leak detector tool. This review will be available from this Blog along with the sample (leaking) Java EE program I created for that purpose.

Plumbr tool creator’s mission

I can assure you that Java memory leaks can be quite complex to solve as there are different memory leak patterns and the analysis phase can be tricky; especially when you attempt to differentiate between leaking Java objects and healthy memory footprint.

That being said, I agree with Plumbr creators that Java memory leak analysis should not be only reserved to Java experts; this is one of the reasons I created this Blog at the first place, for the less experienced Java EE individuals who desire to learn key problem patterns such as Java memory leaks and how to resolve them.

Learning these advanced troubleshooting principles is always easier when we are equipped with the proper tools.

Plumbr’s primary objective is to assist you in your troubleshooting effort by learning about your Java / Java EE application and then to detect potential memory leak source(s) and location(s) within your implementation.

Now let’s begin...

Please bookmark this article and stay tuned for the full review and case study.

Read More