# Heap Allocation Failures on 8.17 **URL:** \ **Category:** Elasticsearch\ **Tags:** runtime-fields\ **Created:** [December 19, 2024, 8:54am UTC](https://discuss.elastic.co/t/heap-allocation-failures-on-8-17/372211 "2024-12-19T08:54:43Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Post date:** [December 19, 2024, 8:54am UTC](https://discuss.elastic.co/t/heap-allocation-failures-on-8-17/372211/1 "2024-12-19T08:54:43Z")
We've had 2 different 8.x clusters running for a number of years without issue however since upgrading to 8.17 (from 8.15.1) a couple of days ago we are seeing frequent heap allocation failures such as below: ```auto # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 1048576 bytes. Error detail: AllocateHeap # An error report file with more information is saved as: # /usr/share/elasticsearch/hs_err_pid514.log ``` The behaviour of searching & indexing patterns is unchanged. From the example above showing the failure to allocate memory, the recorded heap is not anywhere near the limit in the moments leading up to the crash: (the dip is when the pod restarted), which indicates lots of heap being allocated in a very narrow space of time? ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/d/3df6b4588e955340c2e91f1130d9d39814f98950.png)e For further curiosity, it is only our hot nodes (`transform`, `data_hot` & `data_content` are the roles it has that e.g. our warm nodes do not) that are seeing this issue, and also the OOM's are fairly distributed across these hot nodes; each hot node has OOM'd twice since upgrading 2 days ago; and each set of OOM's all seem to happen within a couple of hours of each other These are the running args for the elastic process: ```auto /opt/java/openjdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT -Des.distribution.type=tar --enable-native-access=org.elasticsearch.nativeaccess,org.apache.lucene.core -XX:ReplayDataFile=logs/replay_pid%p.log -Djava.security.manager=allow -Des.cgroups.hierarchy.override=/ -Xms27136m -Xmx27136m -Xlog:async -Xlog:gc*,gc+ergo*=trace,gc+heap=debug,gc+age=trace,gc+metaspace=debug,gc+jni=debug,gc+ihop=debug,safepoint,stringdedup*=debug:file=/atcloud/gclogs/gc-elasticsearch-data-hot-3-2024_12_19-00_05.log:tags,time,level:filecount=3,filesize=10M -server -Xss1m -Xlog:gc+heap+coops=info -XshowSettings:vm -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -XX:-OmitStackTraceInFastThrow -XX:+AlwaysPreTouch -XX:+UnlockExperimentalVMOptions -XX:+UseCondCardMark -XX:-UseSerialGC -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:GCPauseIntervalMillis=1000 -XX:InitiatingHeapOccupancyPercent=60 -XX:+ParallelRefProcEnabled -XX:+UseDynamicNumberOfGCThreads -XX:+PerfDisableSharedMem -XX:+AlwaysActAsServerClassMachine -XX:-ResizePLAB -Des.networkaddress.cache.ttl=30 -Des.networkaddress.cache.negative.ttl=0 -Des.allow_insecure_settings=true -XX:MaxDirectMemorySize=14227079168 -XX:G1ReservePercent=25 --module-path /usr/share/elasticsearch/lib --add-modules=jdk.net --add-modules=ALL-MODULE-PATH -m org.elasticsearch.server/org.elasticsearch.bootstrap.Elasticsearch ``` The Kubernetes pods themselves have 15 CPU with 53Gi memory (request, no explicit limit) The main things I could see in the changelog notes between 8.15 & 8.17 was around the `logsdb` index type but I can confirm we have not created any indices using this type (other than any Elastic managed stuff like APM that may be using them), otherwise we don't use any datastreams. Happy to grab any further information that could be useful, we have recorded metrics around plenty of index stats etc. but I've yet been able to make any correlation to what's happening when these OOM's occur (other than searching/indexing rates looking consistent) Our other smaller cluster (2 hot nodes, 7 CPU, 25Gb memory, 12800m heap) is also displaying the exact same OOM's since uprading to 8.17, and again the heap looks very under utilized when the OOM happens (in this example ~22:30): ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/b/ab25515fd549cd921dc5abf52242040f83460444.png) Edit: I'm now understanding that the memory allocation failure is for memory allocated outside the heap, though I'm still not sure what changes were in 8.16/8.17 that would change the memory profile of these hot nodes ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 19, 2024, 11:15am UTC](https://discuss.elastic.co/t/heap-allocation-failures-on-8-17/372211/2 "2024-12-19T11:15:53Z")
A interesting and very well written question. I am certainly curious how this one turns out. Sadly I dont have any great idea, but did you also update other things at same time, or solely the elasticsearch versions? i.e. maybe OS/docket/k8s updates/patches, or some related tooling (Beats, Logstash, Kibana, ...) ? ---
**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Post date:** [December 19, 2024, 12:00pm UTC](https://discuss.elastic.co/t/heap-allocation-failures-on-8-17/372211/3 "2024-12-19T12:00:19Z")
> Sadly I dont have any great idea, but did you also update other things at same time, or solely the elasticsearch versions? i.e. maybe OS/docket/k8s updates/patches, or some related tooling (Beats, Logstash, Kibana, ...) ? Kibana was upgraded too which is interesting as it's of course possible something has changed in Kibana with regards to how it executes queries or similar (which is were the bulk of our queries come from) However, that is to say the OOM's we've seen happen to be later in the evening when user search activity is at a daily low, but our write indices that live on these hot nodes are busiest both for ingest volume, and size (since they are rolled over at midnight) ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 19, 2024, 1:00pm UTC](https://discuss.elastic.co/t/heap-allocation-failures-on-8-17/372211/4 "2024-12-19T13:00:04Z")
Yeah, was just a wild guess. You could disable kibana access temporarily, and see if it still OOMs, but given what you wrote I'd rate it as unlikely. Good luck, hope someone else fires in a better idea. ---
**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\ **Post date:** [December 19, 2024, 4:10pm UTC](https://discuss.elastic.co/t/heap-allocation-failures-on-8-17/372211/5 "2024-12-19T16:10:29Z")
This sounds to me like a JVM issue. I think the [java version was updated](https://github.com/elastic/elasticsearch/pull/114823) in Elasticsearch 8.16.0, maybe you want to try a different one and see if that fixes the issue? ---
**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Post date:** [December 19, 2024, 5:13pm UTC](https://discuss.elastic.co/t/heap-allocation-failures-on-8-17/372211/6 "2024-12-19T17:13:11Z")
Thanks for the reply @Ignacio_Vera We're not making use of the bundled Java version with Elastic and use our own which won't have changed with the upgrade: ```auto openjdk 21.0.5 2024-10-15 LTS OpenJDK Runtime Environment Temurin-21.0.5+11 (build 21.0.5+11-LTS) OpenJDK 64-Bit Server VM Temurin-21.0.5+11 (build 21.0.5+11-LTS, mixed mode, sharing) ``` The same is true of all the JVM args being used so I think that can be ruled out as a potential culprit ---
**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Post date:** [December 20, 2024, 8:03am UTC](https://discuss.elastic.co/t/heap-allocation-failures-on-8-17/372211/7 "2024-12-20T08:03:34Z")
As a further test I introduced an additional hot node to the cluster to reduce the total # of shards on each node, as well as the amount of writes & reads each hot node would be handling, but the same OOM's have been observed across all hot nodes again, between 20:30 - 01:00 ---
**Author:** ![Chris\_Hegarty](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_hegarty/32/140244_2.png) [@Chris\_Hegarty](https://discuss.elastic.co/u/Chris_Hegarty)\ **Post date:** [December 23, 2024, 10:05am UTC](https://discuss.elastic.co/t/heap-allocation-failures-on-8-17/372211/8 "2024-12-23T10:05:48Z")
> [@Evesy](#): > > Edit: I'm now understanding that the memory allocation failure is for memory allocated outside the heap, though I'm still not sure what changes were in 8.16/8.17 that would change the memory profile of these hot nodes Is there an hs\_err\_pidXXXX.log? If so, would it be possible to post part of it (or put in a gist on GitHub or elsewhere)? Quite often it will have information relating to failures outside of the JVM. For example, Elasticsearch has some native implementation outside of the JVM, but so does the JDK. It would be good to try to narrow down where the failure is occurring. ---
**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Post date:** [December 30, 2024, 10:38am UTC](https://discuss.elastic.co/t/heap-allocation-failures-on-8-17/372211/9 "2024-12-30T10:38:48Z")
@Chris_Hegarty Sorry I forgot to post an update in this thread, only shared it in another related thread: [OOM since 8.16.1 with openjdk23 - #9 by Evesy](https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23/371395/9) Increasing the amount of memory regions allowed to be used by Elastic appears to have resolved the issue for us. Assuming our Elasticsearch nodes were right on the limit prior to upgrading from 8.15 to 8.17 it would appear Elastic is now using at least ~30% more memory regions than it was previously. If this feels like something I should raise as a bug report somewhere please let me know. It's certainly unexpected behaviour for us, we've never breached the previous limit of 260k, but after upgrading (and having to then subsequently increase the limit) we've seen nodes going close to 400k ---
**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Post date:** [January 8, 2025, 3:20pm UTC](https://discuss.elastic.co/t/heap-allocation-failures-on-8-17/372211/10 "2025-01-08T15:20:15Z")
Opened [Elasticsearch 8.16.x Large Increase in MMAP Counts · Issue #119652 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/119652)