Memory issues in production systems Production system Restricted

Memory issues in production systems

Production system Restricted access Application, DB, Application server, log files Debugging, monitoring Investigation should not affect users Reproducing memory issue is not the best idea

Memory leaks What is memory leak java. lang. Out. Of. Memory: heap size other symptoms - extensive garbage collection Hard to discover Hard to investigate Users don't recognize it Few information Sometimes hard to fix Design issues

Investigation of memory issues Log files Monitoring behavior of running application logs, access logs, gc logs JConsole Heap dump Analysis JHAT, MAT, Netbeans

Log files Motivation find issues find patterns leading to issues Problems combining different logs (access log, application log) bad visibility of root cause – not visible or hidden in big amount of minor issues memory problems trigger only few types of messages

Investigating log files Application logs Access logs Log free memory once per minute Analysis of traffic and business use cases GC logs Show problems with GC (stop the world)

Monitoring via JMX Tools JConsole Motivation More information than in logs Monitoring tests Problems Not always applicable on production system (security reasons) Test might not cover all scenarios

Heap dump analysis Tools Motivation JMAP, JHAT, MAT, Netbeans Offline investigation of behavior during specific time frame Comparison of different heap dumps Problems Access to heap dumps (security)

Creating Heap dump jmap -dump: live, format=b, file=heap. dump <pid> Part of JDK Used to create heap dump on demand JVM parameters -XX: +Heap. Dump. On. Out. Of. Memory. Error XX: Heap. Dump. Path=/directory/subdirectory Produces file java_pid<pid>. hprof Used for production system, creates heap dump when Out. Of. Memory is thrown

Tools for heap dump analysis JHAT Part of JDK 6 Running as web application IDE Netbeans, MAT (Eclipse plugin) Commercial tools

Heap dump analysis - JHAT Heap Histogram List of classes with number of instances and used memory Referrers vs Referees Navigating in object graph Example: Form → String → [c

Heap dump analysis – Netbeans, MAT Dominator tree Shallow vs Retained heap Object x dominates object y if every path in object graph from root to y goes through x Shallow heap is memory used by object Retained set of X is set of objects which would be removed by GC if X is removed by GC Retained heap is sum of shallow heap in retained set Other queries Leak suspects

Heap dump Analysis - OQL Object query Language Different dialect for each tool JHAT Simple SQL combined with javascript callback functions MAT More SQL like (Subquery, union, distinct) Predefined functions (dominator)

OQL Examples JHAT select heap. find. Class('org. apache. catalina. util. Server. Info'). statics. server. Number. to. String() MAT SELECT server. Number. to. String() FROM OBJECTS org. apache. catalina. util. Server. Info

Experiences Bugs Incorrect usage of external libraries or frameworks Design issues

Bugs

Usage of frameworks Hibernate Short time usage of big amount of memory Too big first level cache Good understanding of framework is a must

Design Issues Overused session scope Big pages Big buffer in page context Page context referenced by tag handlers Heavy load causes big memory usage Fix might require changing use cases Reusing functionality

Interesting links http: //blogs. sun. com/sundararajan/entry/querying_java_heap_with_oql http: //www. eclipse. org/mat/downloads. php OQL help for JHAT Standalone MAT www. google. com