There are 8 types of OutOfMemoryError. In this, 5 types of OutOfMemoryError happens within the JVM heap.
- Java heap space
- GC overhead limit exceeded
- Requested array size exceeds VM limit
- Permgen space
- Metaspace – Remaining 3 types of OutOfMemoryError happens outside the heap:
- Unable to create new native thread
- 7. Kill process or sacrifice child
- 8. reason stack_trace_with_native_method
Procedures to diagnose and solve all the 5 types of OutOfMemoryErrors that happen within the JVM heap are same. In this article, let’s explore how to diagnose these 5 types of OutOfMemoryErrors.
Tools
One of the immortal quotes from Abraham Lincoln is: “Give me six hours to chop down a tree, and I will spend the first four sharpening the axe”. He says the tool that we choose for the job plays a crucial role towards the success. It’s very true when it comes to diagnosing memory leaks. Thus, here we would like to recommend 2 important tools for diagnosing memory leaks:
- GCeasy
GCeasy is a universal Garbage collection log analyzer, which has the intelligence to analyze and detect memory problems from the garbage collection log files. Using this tool, you will be able to detect whether the application is suffering from memory leaks or not. This is an online tool; you don’t have to download or install it.
- Eclipse MAT
Eclipse MAT is a powerful JVM heap dump analysis tool. There are multiple JVM heap dump analysis tools. In fact, JDK itself ships with a heap dump analyzer called JHAT. But in my personal experience, I have found Eclipse MAT to work best, especially when handling large sized heap dumps. You can download Eclipse MAT from this location. Here are some important Eclipse MAT tit-bits to boost your productivity.
Procedure
Here are the steps to follow to diagnose memory leaks:
Step 0: Validate whether memory is leaking
In the military, there is a quote: “Don’t trust anyone.” This is very much applicable when it comes to diagnosing production problems. Because of the heat of the moment, fellow engineers might come up with a wrong conclusion that memory is leaking while there are no memory problems. So the first step in the diagnosis is to confirm whether memory is actually leaking or not. To confirm it, you need to do 2 things:
- In your log file or standard error, stream grep whether “java.lang.OutOfMemoryError” is printed. When allocated memory is saturated, JVM prints this error message.
2. Upload your Garbage Collection log file toGC Easy. It has the intelligence to figure out whether the memory is leaking in the application or not.
If you get confirmation from either one of the sources, then proceed to next step. If not, your application could be experiencing some other problem and not the memory leak.
Step 1: Capture baseline heap dump
A heap dump is a snapshot of the memory of a Java process. The snapshot contains information about the Java objects and classes in a heap at the moment the snapshot is taken.
You need to capture heap dump from the application when it is in a healthy state. Typically memory leak starts to happen only after few minutes or few hours since the application has started. So after starting your application, let the application take traffic for 10 – 15 minutes, after this point capture the heap dump.
Tit-bit 1:
Here are the step-by-step instructions to capture a heap dump.
Tit-bit 2:
It’s always better to capture heap dump in the production environment (unless in the test environment you can mirror the exact production traffic pattern). Traffic type and it’s volume plays a primary role in the type and amount of objects created in the memory.
Tit-bit 3:
Please be advised that capturing heap dumps takes time, anywhere from few seconds to few minutes based on your heap size. Thus when you are capturing heap dump from the production environment, disable the traffic to the JVM instance where you are capturing heap dump. This will help you to avoid any customer impact.
Step 2: Capture troubled heap dump
Let your application run until it starts to exhibit memory problems. Once memory problem starts to surface, capture heap dump once again. Now the question becomes, how will I know whether the application is exhibiting memory problems or not? Typically when memory problems start to happen, you will notice one of the following symptoms:
- CPU consumption might start to spike
- Application might become unresponsive
- “java.lang.OutOfMemoryError” string will be printed in your application log file or standard error stream
Have proper monitoring/alerting in place. Once any of these symptoms starts to happen, capture heap dumps.
Tit-bit 4:
Sometimes, when application is in the middle of memory problems, you wouldn’t be able to capture heap dumps. In those circumstances, it’s OK (but not ideal) to capture heap dumps right before memory problems starts to appear.
Step 3: Do comparative heap dumps analysis
Upload the heap dump captured in step 1 to Eclipse MAT. Now note down the top 5 objects that are occupying the memory. Most of the time (if not all the time) memory leak surfaces because of one of the top 5 objects.
Now upload the heap dump captured in step 2 to Eclipse MAT. Note down top 5 objects. If any of them shows a significant increase, then that is the object that is leaking. Now you know which the objects that are leaking are. Eclipse MAT has the option to provide you the references of those objects. If you walk through the references, you would be able to figure which Object/method/line of code is causing this memory leak.
In most of the memory leaks, diagnosing is a painful process; fixing it might become very trivial. Equipped with this knowledge, what are the leaking objects and who is creating them, you should be able to fix the root cause of the memory leak.
Every single day millions & millions of people in North America – bank, travel and do commerce using the applications that Ram Lakshmanan has architected. Ram is an acclaimed speaker in major conferences on the scalability, availability, performance topics. Recently he has founded a startup, which specializes in troubleshooting performance problems
