Hello , We have deployed one java service into kubernetes with Heap Size around 4GB. We also run APM along with it using below command . java -javaagent:/home/user/elastic-apm-agent-1.18.0.RC1.jar -jar -Delastic.apm.service_name=A -Delastic.apm.server_urls=http://192.168.x.x:8200 -Delastic.apm.en…

@Ayush_Agrahari thanks for the additional info. Any chance you answer my questions as well? :slight_smile: One of them was how reproducible this is. I ran some long load tests on an Oracle JDK and was not able to reproduce it so far. If it is easily reproducible in your Ubuntu setting, we have a g…

Thanks for reporting! Yes, some more info may help: Please take a look at the heap graphs provided in our APM metrics (JVMs) view in the entire time leading to the OOM error - does it seem like a leak, meaning - continuously increasing heap usage until depleted? Or is it a spike in heap usage sho…

Thanks @Eyal_Koren for replying and providing inputs. Please find information as you asked. [image] Eyal_Koren: Please take a look at the heap graphs provided in our APM metrics (JVMs) view in the entire time leading to the OOM error - does it seem like a leak, meaning - continuously increasi…

Definitely looks like a memory leak :frowning: I will look into that. In the mean time, can you analyze heap references and see what holds all these CallTree objects (which collection in which class)? Thanks!

It looks like it took 3-4 hours before the leak took effect. So a couple more questions for analysis: Did you dynamically change any configuration? Can you find something interesting in logs (agent or app) somewhere between 10PM of the 21st and 2AM of the 22nd? Can you think of anything else that…

[image] Eyal_Koren: Definitely looks like a memory leak :frowning: I will look into that. In the mean time, can you analyze heap references and see what holds all these CallTree objects (which collection in which class)? Thanks! Hey @Eyal_Koren , Sharing more stats related on Heap consumpti…

Thanks @Eyal_Koren . Please find answers on asked questions. [image] Eyal_Koren: Did you dynamically change any configuration? No @Eyal_Koren [image] Eyal_Koren: Can you find something interesting in logs (agent or app) somewhere between 10PM of the 21st and 2AM of the 22nd? Nothing…

[image] Ayush_Agrahari: We get this issue in every 2 to 3 Days. For now As per your suggestion We disabled profiling hence not getting that issue. We can reproduce it in our system. It takes 2-3 Days. This would be extremely useful! Please reenable the sampling profiler on one (or subset) of …

[image] Eyal_Koren: This would be extremely useful! Please reenable the sampling profiler on one (or subset) of your pods but use the agent snapshot provided above. Set the JVM args to provide the most detailed heap dump if and when it crashes. Thanks @Eyal_Koren for quick help and sharing JA…

If not reproduced, please update as well :slight_smile:

APM Consuming lot of HEAP Memory

Elastic Observability APM

Ayush_Agrahari (Ayush Agrahari) January 28, 2021, 7:29am 17

Hi @Eyal_Koren ,

It's been days and we did not find heap issue so far. Thanks for your help

Topic		Replies	Views
APM Consuming lot of reatined Heap and Blocked APM Threads in Thread Dump APM java	7	1029	June 27, 2021
The java application server in the pod was killed due to excessive memory usage APM java	7	607	May 8, 2024
APM Server upgradation to 7.17.6 is consuming more resources APM server	18	1081	February 10, 2023
Java APM1.41.0 JVM used is Zero APM java	8	575	September 7, 2023
Java APM agent not pushing CPU & Memory metric - IBM Java on WebSphere Liberty APM java	7	893	May 11, 2020

APM Consuming lot of HEAP Memory

Related topics