# OOM since 8.16.1 with openjdk23 **URL:** \ **Category:** Elasticsearch\ **Tags:** runtime-fields\ **Created:** [December 3, 2024, 2:39pm UTC](https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23/371395 "2024-12-03T14:39:00Z")\ **Posts on this page:** 13\ **Page:** 2
**Author:** ![ALIT](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@ALIT](https://discuss.elastic.co/u/ALIT)\ **Post date:** [January 9, 2025, 2:50pm UTC](https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23/371395/21 "2025-01-09T14:50:29Z")
> [@ALIT](#): > > I can also share the JVM fatal error log and the last gc logs before the crash. Yep, as mentioned here, I can share it. @Ignacio_Vera: is it safe to post the file as gist? The file has 73MB and as far as I can see, the only "private data" is the hostname. Anything I should consider? As its our company ES, I would not like to share any confidential data. ---
**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\ **Post date:** [January 10, 2025, 11:48am UTC](https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23/371395/22 "2025-01-10T11:48:52Z")
It should be safe. We only need the initial few hundred lines of the file. If you feel more comfortable you can share it in a private message. ---
**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\ **Post date:** [January 13, 2025, 1:31pm UTC](https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23/371395/23 "2025-01-13T13:31:11Z")
We just uncover a bug on the enrich cache introduced in 8.16 where the cache will slowly grow until it exhaust the JVM heap. If you are using enrich and facing OOM, the way to avoid it is to set the setting `enrich.cache_size` back to the previous value of 1000. > > > \### Elasticsearch Version > > 8.16+ > > \### Installed Plugins > > \_No response\_ > > \### Java… Version > > \_bundled\_ > > \### OS Version > > any > > \### Problem Description > > In 8.16 we changed the default for \`enrich.cache\_size\` from \`1000\` to \`1%\` which should allow us to more carefully cache. The trouble is that we don't account for a few things in the cache's weight: > > 1. The cache key. I'm seeing those in the 600 bytes size. > 2. Empty results. > 3. Results use their serialized size but are then stored as \`Map\\` which is usually much larger. > 4. It sure \*looks\* like there isn't any intrinsic cost to a cache entry. Like, for the entry itself. > > When you combine the first two, queries pointing to no documents in the enrich index cost 0 bytes. So, no matter the limit you configure, you \*can\* fill up memory with them. > > But, yeah, I \*think\* we should include the size of the cache key. And Include an estimate of the cost of the List we store in the cache - even if it's empty. And, finally, I think we should uplift the cost of the exploded map. Or we should cache the serialized copy. I'm not entirely sure if it's worth a cost for the cache entry itself. That bit I don't know. > > \### Steps to Reproduce > > See above. > > \### Logs (if relevant) > > @nik9000 has a heap dump of a node the OOMed full of cache entries of empty results lists. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [January 16, 2025, 10:29am UTC](https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23/371395/24 "2025-01-16T10:29:51Z")
Great that a bug was found, A big THANKS to @ALIT and @Evesy for helping uncover it, with not a little troubleshooting effort too. ---
**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Post date:** [January 16, 2025, 12:29pm UTC](https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23/371395/25 "2025-01-16T12:29:56Z")
Thanks @Ignacio_Vera I will look at reverting this value back to `1000` (though I'm not clear on where these settings are actually set ([Set up an enrich processor | Elasticsearch Guide [8.17] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/enrich-setup.html#ingest-enrich-settings)) and then dropping the mmap limit back down too. Is it expected that this would cause issues in our cluster given that we do not make use of any enrichment processors? ---
**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\ **Post date:** [January 16, 2025, 12:41pm UTC](https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23/371395/26 "2025-01-16T12:41:04Z")
No, if you are not using the enrich processor, you should not b affected. I suspect this is something else ---
**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Post date:** [January 20, 2025, 8:43am UTC](https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23/371395/27 "2025-01-20T08:43:27Z")
@Ignacio_Vera I have the crash log from one of our nodes after lowering the limit back to ~200k, this is on a cluster not using ingest. It appears too large for either Gists and this forum doesn't support uploads of that file type, is there a way I can get it over to you/the Elastic team? Happy to trim it down if you can advise exactly which bits of it are important Thanks in advance! ---
**Author:** ![pavlodvornikov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavlodvornikov/32/141457_2.png) [@pavlodvornikov](https://discuss.elastic.co/u/pavlodvornikov)\ **Post date:** [March 27, 2025, 4:07pm UTC](https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23/371395/28 "2025-03-27T16:07:08Z")
Hello, it looks like I'm facing the same issue that I described in [Elasticsearch 8.17.2: Native memory allocation (mmap) failed](https://discuss.elastic.co/t/elasticsearch-8-17-2-native-memory-allocation-mmap-failed/376485) I checked `wc -l /proc/x/maps` on old and upgraded clusters which are loaded more or less the same and the difference is quite big. Old reports `~20k` while new reports `~160k` Were there any recent developments on this topic? ---
**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\ **Post date:** [March 28, 2025, 11:47am UTC](https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23/371395/29 "2025-03-28T11:47:27Z")
Following this comment [Performance degradation after upgrading from 8.6.1 to 8.16.1 · Issue #118623 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/118623#issuecomment-2622802248), would you be able to add `-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=1` to disable grouping of multiple files into one shared segment? ---
**Author:** ![ALIT](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@ALIT](https://discuss.elastic.co/u/ALIT)\ **Post date:** [March 28, 2025, 12:49pm UTC](https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23/371395/30 "2025-03-28T12:49:59Z")
I started one of my nodes with sharedArenaMaxPermits=1. I will monitor map usage and report back on monday. Also started to monitor the number of maps every 30 seconds on all nodes. ---
**Author:** ![ALIT](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@ALIT](https://discuss.elastic.co/u/ALIT)\ **Post date:** [March 31, 2025, 7:13am UTC](https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23/371395/31 "2025-03-31T07:13:49Z")
@Ignacio_Vera - looks like it helped! > **[image hosted at ImgBB](https://ibb.co/9kLV4W53)** > > Image image hosted in ImgBB This is my graph of two different nodes in the cluster, after applying the change on the first (green) node. Funny enough, just some minutes ago, the second node crashed, when it hit the 500k maps 😉 ---
**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\ **Post date:** [March 31, 2025, 8:02am UTC](https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23/371395/32 "2025-03-31T08:02:45Z")
Thank you! Looking at the graph, it does look like a leak but we will need to dig in. ---
**Author:** ![ALIT](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@ALIT](https://discuss.elastic.co/u/ALIT)\ **Post date:** [May 2, 2025, 7:14am UTC](https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23/371395/33 "2025-05-02T07:14:30Z")
Is there any news on this topic? [Previous page](https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23/371395.md?page=1)