# APM Consuming lot of reatined Heap and Blocked APM Threads in Thread Dump **URL:** \ **Category:** APM\ **Tags:** java\ **Created:** [May 24, 2021, 3:10am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-reatined-heap-and-blocked-apm-threads-in-thread-dump/273778 "2021-05-24T03:10:42Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Ayush\_Agrahari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_agrahari/32/74932_2.png) [@Ayush\_Agrahari](https://discuss.elastic.co/u/Ayush_Agrahari)\ **Post date:** [May 24, 2021, 3:10am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-reatined-heap-and-blocked-apm-threads-in-thread-dump/273778/1 "2021-05-24T03:10:42Z")
Hello, We have deployed one java service into Kubernetes with a Heap Size of around 8GB. We also run APM along with it using the below command. ```auto java -javaagent:/home/user/elastic-apm-agent-1.23.0.jar -jar -Delastic.apm.service_name=A -Delastic.apm.server_urls=http://192.168.x.x:8200 -Delastic.apm.environment=Development -Delastic.apm.profiling_inferred_spans_enabled=true -Delastic.apm.enable_log_correlation=true -Delastic.apm.profiling_inferred_spans_excluded_classes=co.elastic.* /home/user/A.war ``` We are using the latest APM Version(1.23.0). We are running our setup in Kubernetes. We found that the Service pod was getting restarted frequently with load testing and the root cause was OOM. **Observation:** It does not seem memory leak as we could not found a memory leak pattern for the long-running pod. It seems more that some processes are taking a lot of Heap when getting load. To Debug Further, we took some Heap Dump of the running pod and we found that APM was consuming more than 40% of Retained Heap, and also APM threads were in the blocked state in Thread Dump. Attaching Heap Dump Snapshots & Required Details to you for debugging further. 113,652,168 bytes (35.77 %) of Java heap is used by 3,071 instances of java/util/concurrent/ConcurrentHashMap$Node co/elastic/apm/agent/shaded/bytebuddy/pool/TypePool$CacheProvider$Simple at 0x7fd1b35a0 ![Screenshot from 2021-05-24 08-39-39](https://us1.discourse-cdn.com/elastic/original/3X/b/0/b03d99cbfdedb396909e150a403b9678e5fc4da6.jpeg) JVM & OS: ![Screenshot from 2021-05-24 08-35-45](https://us1.discourse-cdn.com/elastic/original/3X/1/7/1782d31e3540e1db8479c84d38c71162a9cba6b0.png) Could you please help me with this. Let me know if need any information. ---
**Author:** ![Ayush\_Agrahari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_agrahari/32/74932_2.png) [@Ayush\_Agrahari](https://discuss.elastic.co/u/Ayush_Agrahari)\ **Post date:** [May 24, 2021, 3:31am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-reatined-heap-and-blocked-apm-threads-in-thread-dump/273778/2 "2021-05-24T03:31:06Z")
@Eyal_Koren Could you or someone from APM team please help here. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [May 26, 2021, 10:08am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-reatined-heap-and-blocked-apm-threads-in-thread-dump/273778/3 "2021-05-26T10:08:44Z")
Hi @Ayush_Agrahari 👋 Great analysis, very useful! Unless you want to increase the heap size, you would need to disable type-pool caching by setting the `enable_type_pool_cache` config option to `false`. It is not a documented one, because it is very rarely changed from the default. The way to set it would be one of the following: - through an agent config file - `enable_type_pool_cache=false` - as a system property in the command line: `-Delastic.apm.enable_type_pool_cache=false` - as an environment variable: `ELASTIC_APM_ENABLE_TYPE_POOL_CACHE=false` It may make startup time longer, but based on your configured heap, I assume this is not a huge app, so it wouldn't necessarily be an issue. I hope this helps. ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [May 27, 2021, 6:54am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-reatined-heap-and-blocked-apm-threads-in-thread-dump/273778/4 "2021-05-27T06:54:44Z")
Also note that the type cache is cleared after it hasn't been accessed since a minute. Usually, the type pool cache is only used on startup. After your app has warmed up, the cache should be automatically cleared. Also, the cache is referenced via `SoftReference`s so that they get cleared automatically if the JVM heap usage approaches the limit. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [May 31, 2021, 6:25am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-reatined-heap-and-blocked-apm-threads-in-thread-dump/273778/5 "2021-05-31T06:25:21Z")
As @felixbarny noted, this cache should not cause OOM, and it should not consume any heap after some up time of the application. We think we know why this wasn't behaving as expected. Please try to use [this fix snapshot](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-java%2Fapm-agent-java-mbp/detail/PR-1828/12/artifacts), without the `enable_type_pool_cache` config, and let us know if this resolves the issue. Thanks! ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [June 1, 2021, 2:37am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-reatined-heap-and-blocked-apm-threads-in-thread-dump/273778/6 "2021-06-01T02:37:30Z")
Version [1.24.0](https://www.elastic.co/guide/en/apm/agent/java/current/release-notes-1.x.html#release-notes-1.24.0) has been released with this fix. ---
**Author:** ![Ayush\_Agrahari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_agrahari/32/74932_2.png) [@Ayush\_Agrahari](https://discuss.elastic.co/u/Ayush_Agrahari)\ **Post date:** [June 6, 2021, 12:42pm UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-reatined-heap-and-blocked-apm-threads-in-thread-dump/273778/7 "2021-06-06T12:42:14Z")
Thanks @Eyal_Koren @felixbarny . I will try this and will update on Consuming Heap Part. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 27, 2021, 8:42am UTC](https://discuss.elastic.co/t/apm-consuming-lot-of-reatined-heap-and-blocked-apm-threads-in-thread-dump/273778/8 "2021-06-27T08:42:53Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.