# APM Consuming lot of HEAP Memory

**URL:** <https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571>\
**Category:** APM\
**Tags:** java\
**Created:** [December 24, 2020, 9:17am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571 "2020-12-24T09:17:51Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ayush\_Agrahari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_agrahari/32/74932_2.png) [@Ayush\_Agrahari](https://discuss.elastic.co/u/Ayush_Agrahari)\
**Post date:** [December 24, 2020, 9:17am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/1 "2020-12-24T09:17:52Z")

</div>

Hello ,

We have deployed one java service into kubernetes with Heap Size around 4GB. We also run APM along with it using below command  
.

```auto
java -javaagent:/home/user/elastic-apm-agent-1.18.0.RC1.jar -jar -Delastic.apm.service_name=A -Delastic.apm.server_urls=http://192.168.x.x:8200 -Delastic.apm.environment=Integration -Delastic.apm.profiling_inferred_spans_enabled=true -Delastic.apm.enable_log_correlation=true -Delastic.apm.profiling_inferred_spans_excluded_classes=co.elastic.* /home/user/A.war

```

We started getting pod out Of Memory Errors in every 2-3 Days. To debug We took heap dump and found co.elastic.apm.agent.profiler.CallTree [2 Million objects] Object itself was consuming more than 2.5GB Heap Memory Space and eventually pod was going OutOfMeory.

```auto
co.elastic.apm.agent.profiler.CallTree [2 Million objects]

```

Could you please help me on this. Let me know if need any information.

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [December 24, 2020, 1:37pm UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/2 "2020-12-24T13:37:50Z")

</div>

Thanks for reporting!  
Yes, some more info may help:

1. Please take a look at the heap graphs provided in our APM metrics (JVMs) view in the entire time leading to the OOM error - does it seem like a leak, meaning - continuously increasing heap usage until depleted? Or is it a spike in heap usage shortly before the crash?
2. If possible, please provide such heap for analysis
3. Provide as many details as possible on these pods - exact OS, exact JVM version etc.
4. If there are any other configurations you make through env variables or otherwise, please provide them as well.

Until we get to the bottom of this, you can run your app without setting the `-Delastic.apm.profiling_inferred_spans_enabled` config to avoid these crashes.

Also, please upgrade to the latest agent version so we know we are not looking into something that is already irrelevant.

Thanks!

---

<div class="post-metadata">

**Author:** ![Ayush\_Agrahari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_agrahari/32/74932_2.png) [@Ayush\_Agrahari](https://discuss.elastic.co/u/Ayush_Agrahari)\
**Post date:** [December 25, 2020, 4:15am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/3 "2020-12-25T04:15:08Z")

</div>

Thanks @Eyal_Koren for replying and providing inputs.

Please find information as you asked.

> [@Eyal\_Koren](#):
>
> 1. Please take a look at the heap graphs provided in our APM metrics (JVMs) view in the entire time leading to the OOM error - does it seem like a leak, meaning - continuously increasing heap usage until depleted? Or is it a spike in heap usage shortly before the crash?

It's memory leak. Continously increasing heap usage until depleted.

 ![Screenshot from 2020-12-25 09-18-27](https://us1.discourse-cdn.com/elastic/original/3X/7/c/7c7a0a1cfddd129d6f217a7e59188d12b90c8068.png)

> [@Eyal\_Koren](#):
>
> 1. If possible, please provide such heap for analysis

It would not be possible to share heap for analysis. I am sharing screenshot of analysis. Please let me know what steps need to be followed in analysis. Would do and provide you info.

 ![Screenshot from 2020-12-25 09-39-20](https://us1.discourse-cdn.com/elastic/original/3X/4/1/413f061c6c3dd0ee993217780a7ed492c367ac78.png)

> [@Eyal\_Koren](#):
>
> 1. Provide as many details as possible on these pods - exact OS, exact JVM version etc.

I am sharing JVM and OS Version information below.

 ![Screenshot from 2020-12-25 09-38-21](https://us1.discourse-cdn.com/elastic/original/3X/8/3/83cc67dad4ab43ed9f4810badb4335998fe9a9df.png)

 ![Screenshot from 2020-12-25 19-28-28](https://us1.discourse-cdn.com/elastic/original/3X/3/d/3d12f0961d20d13d3d69faea476cbd4481f86452.png)

> [@Eyal\_Koren](#):
>
> 1. If there are any other configurations you make through env variables or otherwise, please provide them as well.

No Other configuration. I am sharing config file of APM Server below.

```auto
apm-server:
  # Defines the host and port the server is listening on. Use "unix:/path/to.sock" to listen on a unix domain socket.
  host: "0.0.0.0:8200"
  rum.enabled: true
  kibana:
    enabled: true
    host: "https://${KIBANA_HOST}:${KIBANA_PORT}"
    #protocol: "https"
    username: ${ELASTIC_USERNAME}
    password: ${ELASTIC_PASSWORD}
    ssl.enabled: true
    ssl.verification_mode: none
    ssl.certificate_authorities: ["/certs/ca.crt"]

monitoring:
  enabled: true

output.elasticsearch:
  hosts: ["https://${ELASTICSEARCH_HOST_PORT}"]
  username: ${ELASTIC_USERNAME}
  password: ${ELASTIC_PASSWORD}
  ssl.enabled: true
  ssl.certificate_authorities: ["/certs/ca.crt"]

output.elasticsearch.bulk_max_size: 1500
output.elasticsearch.worker: 3
queue.mem.events: 4500

```

We are now upgrading to latest version and have disabled -Delastic.apm.profiling\_inferred\_spans\_enabled for now.

Please let me know if you need any more information.

Thanks,  
Ayush A.

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [December 27, 2020, 7:22am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/4 "2020-12-27T07:22:03Z")

</div>

Definitely looks like a memory leak ☹  
I will look into that. In the mean time, can you analyze heap references and see what holds all these `CallTree` objects (which collection in which class)?  
Thanks!

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [December 27, 2020, 11:06am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/5 "2020-12-27T11:06:24Z")

</div>

It looks like it took 3-4 hours before the leak took effect.  
So a couple more questions for analysis:

1. Did you dynamically change any configuration?
2. Can you find something interesting in logs (agent or app) somewhere between 10PM of the 21st and 2AM of the 22nd?
3. Can you think of anything else that changed or happened during this time?
4. Do you redeploy/stop/start the application running on this JVM (without restarting the entire JVM/Servlet-container/pod) from time to time?
5. How reproducible is this behaviour? If you run the same settings again, would it look the same?
6. Anything else interesting you noticed in the metrics tab? For example, did the number of threads remain stable?

Thanks.

---

<div class="post-metadata">

**Author:** ![Ayush\_Agrahari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_agrahari/32/74932_2.png) [@Ayush\_Agrahari](https://discuss.elastic.co/u/Ayush_Agrahari)\
**Post date:** [December 28, 2020, 3:29am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/6 "2020-12-28T03:29:07Z")

</div>

> [@Eyal\_Koren](#):
>
> Definitely looks like a memory leak ☹  
> I will look into that. In the mean time, can you analyze heap references and see what holds all these `CallTree` objects (which collection in which class)?  
> Thanks!

Hey @Eyal_Koren,

Sharing more stats related on Heap consumption by CallTree Objects.

 ![Screenshot from 2020-12-28 08-58-03](https://us1.discourse-cdn.com/elastic/original/3X/d/c/dcdd0121ba4f403fe5ea2b87798d531895e372c0.png)  
 ![Screenshot from 2020-12-28 08-52-53](https://us1.discourse-cdn.com/elastic/original/3X/e/c/ec6d6b389ce9b8d38091f90a87fca1a180cd1d93.png) ![Screenshot from 2020-12-28 08-51-26](https://us1.discourse-cdn.com/elastic/original/3X/3/9/390d5280e03f1b61ff0b9059c06870ba70411ebe.png)

Thread Stack.

```auto
elastic-apm-sampling-profiler
  at jdk.internal.misc.Unsafe.park(ZJ)V (Native Method)
  at java.util.concurrent.locks.LockSupport.parkNanos(J)V (LockSupport.java:357)
  at co.elastic.apm.agent.profiler.SamplingProfiler.consumeActivationEventsFromRingBufferAndWriteToFile(Lco/elastic/apm/agent/configuration/converter/TimeDuration;)V (SamplingProfiler.java:395)
  at co.elastic.apm.agent.profiler.SamplingProfiler.profile(Lco/elastic/apm/agent/configuration/converter/TimeDuration;Lco/elastic/apm/agent/configuration/converter/TimeDuration;)V (SamplingProfiler.java:346)
  at co.elastic.apm.agent.profiler.SamplingProfiler.run()V (SamplingProfiler.java:317)
  at java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; (Executors.java:515)
  at java.util.concurrent.FutureTask.run()V (FutureTask.java:264)
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V (ScheduledThreadPoolExecutor.java:304)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1128)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:628)
  at java.lang.Thread.run()V (Thread.java:834)

```

![Screenshot from 2020-12-28 09-11-01](https://us1.discourse-cdn.com/elastic/original/3X/e/5/e5408f0bd0e06d5646e374f60762501ea38e5dfb.png)

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [December 28, 2020, 6:36am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/7 "2020-12-28T06:36:45Z")

</div>

@Ayush_Agrahari thanks for the additional info.  
Any chance you answer my questions as well? 🙂

One of them was how reproducible this is. I ran some long load tests on an Oracle JDK and was not able to reproduce it so far. If it is easily reproducible in your Ubuntu setting, we have a good place to test it. Can you please try [this snapshot](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-java%2Fapm-agent-java-mbp/detail/PR-1592/1/artifacts) and see if it resolves the problem? I did some initial analysis of the code and made some small changes in Object recycling.

---

<div class="post-metadata">

**Author:** ![Ayush\_Agrahari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_agrahari/32/74932_2.png) [@Ayush\_Agrahari](https://discuss.elastic.co/u/Ayush_Agrahari)\
**Post date:** [December 28, 2020, 6:43am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/8 "2020-12-28T06:43:51Z")

</div>

Thanks @Eyal_Koren . Please find answers on asked questions.

> [@Eyal\_Koren](#):
>
> Did you dynamically change any configuration?

No @Eyal_Koren

> [@Eyal\_Koren](#):
>
> 1. Can you find something interesting in logs (agent or app) somewhere between 10PM of the 21st and 2AM of the 22nd?

Nothing interesting I found.

> [@Eyal\_Koren](#):
>
> 1. Can you think of anything else that changed or happened during this time?

No, As this error we are getting repeatedly in every 3-4 Days.

> [@Eyal\_Koren](#):
>
> 1. Do you redeploy/stop/start the application running on this JVM (without restarting the entire JVM/Servlet-container/pod) from time to time?

We redeploy pod it means old pod would be destroyed and new Pod JVM would be deployed.

> [@Eyal\_Koren](#):
>
> 1. How reproducible is this behaviour? If you run the same settings again, would it look the same?

We get this issue in every 2 to 3 Days. For now As per your suggestion We disabled profiling hence not getting that issue. We can reproduce it in our system. It takes 2-3 Days.

> [@Eyal\_Koren](#):
>
> 1. Anything else interesting you noticed in the metrics tab? For example, did the number of threads remain stable?

Nothing as such. During last 1-3 hour before OOM Thread count got increased by 5-10%.

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [December 28, 2020, 6:55am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/9 "2020-12-28T06:55:13Z")

</div>

> [@Ayush\_Agrahari](#):
>
> We get this issue in every 2 to 3 Days. For now As per your suggestion We disabled profiling hence not getting that issue. We can reproduce it in our system. It takes 2-3 Days.

This would be extremely useful! Please reenable the sampling profiler on one (or subset) of your pods but use the agent snapshot provided above. Set the JVM args to provide the most detailed heap dump if and when it crashes.

If this agent snapshot fixes the issue - great. If not, please try again to analyse your heap dump as you did, to figure out exactly which field/collection of the `SamplingProfiler` instance is holding these huge `CallTree$Root` objects (assuming you cannot provide the heap dump itself).

Thanks!

---

<div class="post-metadata">

**Author:** ![Ayush\_Agrahari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_agrahari/32/74932_2.png) [@Ayush\_Agrahari](https://discuss.elastic.co/u/Ayush_Agrahari)\
**Post date:** [December 28, 2020, 7:25am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/10 "2020-12-28T07:25:55Z")

</div>

> [@Eyal\_Koren](#):
>
> This would be extremely useful! Please reenable the sampling profiler on one (or subset) of your pods but use the agent snapshot provided above. Set the JVM args to provide the most detailed heap dump if and when it crashes.

Thanks @Eyal_Koren for quick help and sharing JAR link.

I will update APM Agent and would observe JVM for OOM Error. If we get again OOM then would share stats with you.

Thanks Again,

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [December 28, 2020, 7:27am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/11 "2020-12-28T07:27:54Z")

</div>

If not reproduced, please update as well 🙂

---

<div class="post-metadata">

**Author:** ![Ayush\_Agrahari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_agrahari/32/74932_2.png) [@Ayush\_Agrahari](https://discuss.elastic.co/u/Ayush_Agrahari)\
**Post date:** [December 28, 2020, 7:35am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/12 "2020-12-28T07:35:35Z")

</div>

Sure @Eyal_Koren

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [January 3, 2021, 7:23am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/13 "2021-01-03T07:23:14Z")

</div>

Hey @Ayush_Agrahari, did you get the chance to test the fix?

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [January 6, 2021, 7:18am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/14 "2021-01-06T07:18:57Z")

</div>

@Ayush_Agrahari one last ping - we are about to merge the related PR that contains the proposed fix, so it would be nice to get your feedback on it.  
Thanks 🙏

---

<div class="post-metadata">

**Author:** ![Ayush\_Agrahari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_agrahari/32/74932_2.png) [@Ayush\_Agrahari](https://discuss.elastic.co/u/Ayush_Agrahari)\
**Post date:** [January 12, 2021, 6:31am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/15 "2021-01-12T06:31:14Z")

</div>

@Eyal_Koren Apology for late reply. We did not get OOM issue with patch version. We started using patch version from last week due to monthly deployment cycle. Will udpate you once we get same issue.

Thanks 🙏

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [January 12, 2021, 7:39am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/16 "2021-01-12T07:39:07Z")

</div>

Thanks for the update @Ayush_Agrahari, this is very encouraging!  
Please try to remember to provide another update in a week or so, if possible.  
Also, you should be aware that this fix was released with [version 1.20.0](https://www.elastic.co/guide/en/apm/agent/java/current/release-notes-1.x.html#release-notes-1.20.0), so you can upgrade to it in your next update cycle.

---

<div class="post-metadata">

**Author:** ![Ayush\_Agrahari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_agrahari/32/74932_2.png) [@Ayush\_Agrahari](https://discuss.elastic.co/u/Ayush_Agrahari)\
**Post date:** [January 28, 2021, 7:29am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/17 "2021-01-28T07:29:35Z")

</div>

Hi @Eyal_Koren ,

It's been days and we did not find heap issue so far. Thanks for your help

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 18, 2021, 3:29am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-heap-memory/259571/18 "2021-02-18T03:29:36Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.