Elasticsearch G1 GC: “To-space Exhausted” and Eden Not Cleared Leading to Potential OOM

[VietBui]

Hi Elasticsearch Community,

I’m running Elasticsearch in a Docker container (version 6.8.22), 3 nodes, each node with a 14GB heap using the G1 garbage collector. My GC logs show issues with “To-space exhausted” and Eden regions not being cleared, which I suspect is causing memory pressure and risking OutOfMemory (OOM) errors.

GC Log Details (GC(392), Young GC):

• Eden: Cleared from 1,025 regions to 0 (97 allocated for next cycle).

• Survivor: 50 → 135 regions, ~1.08 GB live objects (age table).

• Old: 102 → 892 regions, likely due to “To-space exhausted” (Survivor space ran out).

• Pause Time: 445.889ms, exceeding 200ms MMU target.

• Heap: 8,887 MB used post-GC (~62% of 14,336 MB).

• Region Size: Estimated ~8 MB (based on heap and region counts, not explicitly set).

Later GC Issue: In a subsequent GC with ~1,025 Eden regions, Eden was not cleared, and it seems most objects were promoted to the old generation. This differs from GC(392), where Eden was fully cleared. I suspect this is due to a mixed/full GC, allocation failure, or repeated “To-space exhausted.” The rapid old generation growth (892 regions, ~7,136 MB) and high heap occupancy (~62%) suggest a risk of OOM if the old generation fills up.

Setup:

• JVM Options: -Xms14g -Xmx14g -XX:+UseG1GC
• Cluster stats: 240 active shards, 120 primary shards

Questions:

1. Why is Eden not cleared in the later GC, and why are objects being promoted to the old generation, increasing OOM risk?

2. How can I resolve “To-space exhausted” to prevent premature promotions and reduce memory pressure?

3. Should I adjust -XX:G1HeapRegionSize (e.g., 4 MB or 16 MB) or -XX:G1NewSizePercent (currently default?) for a 14 GB heap to avoid OOM?

4. Are there Elasticsearch-specific optimizations (e.g., shard count, indexing settings) to reduce allocation rates and GC pressure?

5. What GC logging options or metrics should I monitor to diagnose potential OOM causes?

[RainTown (Kevin Maguire)]

Welcome to the forum!

As all I know about your elasticsearch setup is limited to:

@VietBui

I’m running Elasticsearch in a Docker container (version 6.8.22), 3 nodes, each node with a 14GB heap using the G1 garbage collector.
Cluster stats: 240 active shards, 120 primary shards

then

@VietBui

Are there Elasticsearch-specific optimizations (e.g., shard count, indexing settings) to reduce allocation rates and GC pressure?

is likely gonna be hard to answer.

btw how much memory is allocated to the containers, (ie. some number hopefully bigger than 14g)

I do like caution, but you said "which I suspect is causing memory pressure and risking OutOfMemory (OOM) errors" - are you actually getting any OOM errors, or you are just worried that they might come? Are the "To-space exhausted" message new/recent, or were they always there?

[Christian_Dahlqvist]

@VietBui

I’m running Elasticsearch in a Docker container (version 6.8.22)

This version is very, very old and has been EOL a very long time. I would recommend that you upgrade to a supported version with urgency.

Which JVM version are you using? I recall G1 GC not being supported in Elasticsearch for a very long time as there were issues related to how it worked under load. I do unfortunately not recall exactly from which JVM version it became supported, but believe it was not supported for Java 8 (if that happens to be what you are using).

[VietBui]

Hi @RainTown

I started the from Out-Of-Memory error on my elasticsearch cluster, it make ES cluster restart several times in day. I searched for the cause and during the investigation of gc log I found some unusual thing about Eden, Suvivor and Old regions as posted above.

I share more detail about my cluster

• 3 nodes Elasticsearch, 32G RAM each hosts with other service running consuming about 4GB Ram, so I configured the heap is 14GB.
• I monitor cluster status of grafana:
  • Docs Count is 58.6 Mil
  • Active Shard 224, active primary shard 113
  • Stored data size is 776gb
  • Indexing rate by node ~ 70
  • Search rate by node ~ 20
• I attach full gc logs for more detail, you can see “to-space exhausted” then after old regions could not be cleaned.

Question:

• what should I do next to investigate the root cause?
• Is there any workaround to prevent this problem?

[Christian_Dahlqvist]

@VietBui

what should I do next to investigate the root cause?

It would be useful if you could let us know which JVM version you are using.

@VietBui

Is there any workaround to prevent this problem?

We do not yet know what the problem is so that is hard to tell. Without any further information I would recommend upgrading or simply switching from G1 to CMS GC and see if that works better.

[VietBui]

Hi @Christian_Dahlqvist

I check in Elasticsearch container Java version is 15.0.1

Gc logs, to space exhausted is in GC 362

[2025-07-23T15:36:49.091+0000][1][gc,age ] GC(360) Desired survivor size 752877568 bytes, new threshold 15 (max threshold 15)

[2025-07-23T15:36:49.113+0000][1][gc,age ] GC(360) Age table with threshold 15 (max threshold 15)

[2025-07-23T15:36:49.113+0000][1][gc,age ] GC(360) - age 1: 55802856 bytes, 55802856 total

[2025-07-23T15:36:49.113+0000][1][gc,age ] GC(360) - age 2: 51612984 bytes, 107415840 total

[2025-07-23T15:36:49.113+0000][1][gc,age ] GC(360) - age 3: 2193312 bytes, 109609152 total

[2025-07-23T15:36:49.113+0000][1][gc,age ] GC(360) - age 4: 68973760 bytes, 178582912 total

[2025-07-23T15:36:49.113+0000][1][gc,age ] GC(360) - age 5: 21374688 bytes, 199957600 total

[2025-07-23T15:36:49.113+0000][1][gc,age ] GC(360) - age 6: 60986776 bytes, 260944376 total

[2025-07-23T15:36:49.113+0000][1][gc,age ] GC(360) - age 7: 24191048 bytes, 285135424 total

[2025-07-23T15:36:49.113+0000][1][gc,age ] GC(360) - age 8: 497112 bytes, 285632536 total

[2025-07-23T15:36:49.113+0000][1][gc,age ] GC(360) - age 9: 86952 bytes, 285719488 total

[2025-07-23T15:36:49.113+0000][1][gc,age ] GC(360) - age 10: 79768 bytes, 285799256 total

[2025-07-23T15:36:49.113+0000][1][gc,age ] GC(360) - age 11: 216432 bytes, 286015688 total

[2025-07-23T15:36:49.113+0000][1][gc,age ] GC(360) - age 12: 390632 bytes, 286406320 total

[2025-07-23T15:36:49.113+0000][1][gc,age ] GC(360) - age 13: 129144 bytes, 286535464 total

[2025-07-23T15:36:49.113+0000][1][gc,age ] GC(360) - age 14: 20616 bytes, 286556080 total

[2025-07-23T15:36:49.113+0000][1][gc,age ] GC(360) - age 15: 75040 bytes, 286631120 total

[2025-07-23T15:36:49.114+0000][1][gc,phases ] GC(360) Pre Evacuate Collection Set: 0.5ms

[2025-07-23T15:36:49.114+0000][1][gc,phases ] GC(360) Merge Heap Roots: 0.3ms

[2025-07-23T15:36:49.114+0000][1][gc,phases ] GC(360) Evacuate Collection Set: 18.0ms

[2025-07-23T15:36:49.114+0000][1][gc,phases ] GC(360) Post Evacuate Collection Set: 3.3ms

[2025-07-23T15:36:49.114+0000][1][gc,phases ] GC(360) Other: 0.5ms

[2025-07-23T15:36:49.114+0000][1][gc,heap ] GC(360) Eden regions: 2082->0(2079)

[2025-07-23T15:36:49.114+0000][1][gc,heap ] GC(360) Survivor regions: 68->71(359)

[2025-07-23T15:36:49.114+0000][1][gc,heap ] GC(360) Old regions: 181->181

[2025-07-23T15:36:49.114+0000][1][gc,heap ] GC(360) Archive regions: 2->2

[2025-07-23T15:36:49.114+0000][1][gc,heap ] GC(360) Humongous regions: 229->155

[2025-07-23T15:36:49.114+0000][1][gc,metaspace] GC(360) Metaspace: 93526K(96384K)->93526K(96384K) NonClass: 82347K(84404K)->82347K(84404K) Class: 11178K(11980K)->11178K(11980K)

[2025-07-23T15:36:49.114+0000][1][gc ] GC(360) Pause Young (Normal) (G1 Evacuation Pause) 10239M->1627M(14336M) 22.638ms

[2025-07-23T15:36:49.114+0000][1][gc,cpu ] GC(360) User=0.25s Sys=0.01s Real=0.02s

[2025-07-23T15:36:58.360+0000][1][gc,start ] GC(361) Pause Young (Normal) (G1 Evacuation Pause)

[2025-07-23T15:36:58.360+0000][1][gc,task ] GC(361) Using 13 workers of 13 for evacuation

[2025-07-23T15:36:58.360+0000][1][gc,age ] GC(361) Desired survivor size 752877568 bytes, new threshold 15 (max threshold 15)

[2025-07-23T15:36:58.602+0000][1][gc,age ] GC(361) Age table with threshold 15 (max threshold 15)

[2025-07-23T15:36:58.602+0000][1][gc,age ] GC(361) - age 1: 1239928568 bytes, 1239928568 total

[2025-07-23T15:36:58.602+0000][1][gc,age ] GC(361) - age 2: 33457160 bytes, 1273385728 total

[2025-07-23T15:36:58.602+0000][1][gc,age ] GC(361) - age 3: 51462376 bytes, 1324848104 total

[2025-07-23T15:36:58.602+0000][1][gc,age ] GC(361) - age 4: 2190248 bytes, 1327038352 total

[2025-07-23T15:36:58.602+0000][1][gc,age ] GC(361) - age 5: 68973696 bytes, 1396012048 total

[2025-07-23T15:36:58.602+0000][1][gc,age ] GC(361) - age 6: 21374544 bytes, 1417386592 total

[2025-07-23T15:36:58.602+0000][1][gc,age ] GC(361) - age 7: 60986680 bytes, 1478373272 total

[2025-07-23T15:36:58.602+0000][1][gc,age ] GC(361) - age 8: 24190920 bytes, 1502564192 total

[2025-07-23T15:36:58.602+0000][1][gc,age ] GC(361) - age 9: 495288 bytes, 1503059480 total

[2025-07-23T15:36:58.602+0000][1][gc,age ] GC(361) - age 10: 85512 bytes, 1503144992 total

[2025-07-23T15:36:58.602+0000][1][gc,age ] GC(361) - age 11: 79768 bytes, 1503224760 total

[2025-07-23T15:36:58.602+0000][1][gc,age ] GC(361) - age 12: 216368 bytes, 1503441128 total

[2025-07-23T15:36:58.602+0000][1][gc,age ] GC(361) - age 13: 390536 bytes, 1503831664 total

[2025-07-23T15:36:58.602+0000][1][gc,age ] GC(361) - age 14: 129144 bytes, 1503960808 total

[2025-07-23T15:36:58.602+0000][1][gc,age ] GC(361) - age 15: 20584 bytes, 1503981392 total

[2025-07-23T15:36:58.602+0000][1][gc,mmu ] GC(361) MMU target violated: 201.0ms (200.0ms/201.0ms)

[2025-07-23T15:36:58.602+0000][1][gc,phases ] GC(361) Pre Evacuate Collection Set: 0.4ms

[2025-07-23T15:36:58.602+0000][1][gc,phases ] GC(361) Merge Heap Roots: 0.2ms

[2025-07-23T15:36:58.602+0000][1][gc,phases ] GC(361) Evacuate Collection Set: 238.2ms

[2025-07-23T15:36:58.602+0000][1][gc,phases ] GC(361) Post Evacuate Collection Set: 2.6ms

[2025-07-23T15:36:58.602+0000][1][gc,phases ] GC(361) Other: 0.8ms

[2025-07-23T15:36:58.602+0000][1][gc,heap ] GC(361) Eden regions: 2079->0(1441)

[2025-07-23T15:36:58.602+0000][1][gc,heap ] GC(361) Survivor regions: 71->359(359)

[2025-07-23T15:36:58.602+0000][1][gc,heap ] GC(361) Old regions: 181->457

[2025-07-23T15:36:58.602+0000][1][gc,heap ] GC(361) Archive regions: 2->2

[2025-07-23T15:36:58.602+0000][1][gc,heap ] GC(361) Humongous regions: 155->155

[2025-07-23T15:36:58.602+0000][1][gc,metaspace] GC(361) Metaspace: 93535K(96384K)->93535K(96384K) NonClass: 82356K(84404K)->82356K(84404K) Class: 11178K(11980K)->11178K(11980K)

[2025-07-23T15:36:58.602+0000][1][gc ] GC(361) Pause Young (Normal) (G1 Evacuation Pause) 9943M->3886M(14336M) 242.297ms

[2025-07-23T15:36:58.602+0000][1][gc,cpu ] GC(361) User=1.89s Sys=0.73s Real=0.24s

[2025-07-23T15:36:58.865+0000][1][gc,start ] GC(362) Pause Young (Normal) (G1 Evacuation Pause)

[2025-07-23T15:36:58.865+0000][1][gc,task ] GC(362) Using 13 workers of 13 for evacuation

[2025-07-23T15:36:58.865+0000][1][gc,age ] GC(362) Desired survivor size 629145600 bytes, new threshold 1 (max threshold 15)

[2025-07-23T15:36:59.199+0000][1][gc,age ] GC(362) Age table with threshold 1 (max threshold 15)

[2025-07-23T15:36:59.199+0000][1][gc,age ] GC(362) - age 1: 1256248456 bytes, 1256248456 total

[2025-07-23T15:36:59.199+0000][1][gc,mmu ] GC(362) MMU target violated: 201.0ms (200.0ms/201.0ms)

[2025-07-23T15:36:59.199+0000][1][gc ] GC(362) To-space exhausted

[2025-07-23T15:36:59.199+0000][1][gc,phases ] GC(362) Pre Evacuate Collection Set: 0.4ms

[2025-07-23T15:36:59.199+0000][1][gc,phases ] GC(362) Merge Heap Roots: 1.2ms

[2025-07-23T15:36:59.199+0000][1][gc,phases ] GC(362) Evacuate Collection Set: 299.3ms

[2025-07-23T15:36:59.199+0000][1][gc,phases ] GC(362) Post Evacuate Collection Set: 33.3ms

[2025-07-23T15:36:59.199+0000][1][gc,phases ] GC(362) Other: 0.5ms

[2025-07-23T15:36:59.199+0000][1][gc,heap ] GC(362) Eden regions: 1441->0(208)

[2025-07-23T15:36:59.199+0000][1][gc,heap ] GC(362) Survivor regions: 359->300(300)

[2025-07-23T15:36:59.199+0000][1][gc,heap ] GC(362) Old regions: 457->2078

[2025-07-23T15:36:59.199+0000][1][gc,heap ] GC(362) Archive regions: 2->2

[2025-07-23T15:36:59.199+0000][1][gc,heap ] GC(362) Humongous regions: 158->158

[2025-07-23T15:36:59.199+0000][1][gc,metaspace] GC(362) Metaspace: 93540K(96384K)->93540K(96384K) NonClass: 82360K(84404K)->82360K(84404K) Class: 11180K(11980K)->11180K(11980K)

[2025-07-23T15:36:59.199+0000][1][gc ] GC(362) Pause Young (Normal) (G1 Evacuation Pause) 9662M->10147M(14336M) 334.659ms

[2025-07-23T15:36:59.199+0000][1][gc,cpu ] GC(362) User=4.17s Sys=0.04s Real=0.33s

[2025-07-23T15:36:59.244+0000][1][gc,start ] GC(363) Pause Young (Concurrent Start) (G1 Evacuation Pause)

[2025-07-23T15:36:59.244+0000][1][gc,task ] GC(363) Using 13 workers of 13 for evacuation

[2025-07-23T15:36:59.244+0000][1][gc,age ] GC(363) Desired survivor size 178257920 bytes, new threshold 1 (max threshold 15)

[2025-07-23T15:36:59.326+0000][1][gc,age ] GC(363) Age table with threshold 1 (max threshold 15)

[2025-07-23T15:36:59.326+0000][1][gc,age ] GC(363) - age 1: 355992584 bytes, 355992584 total

[2025-07-23T15:36:59.326+0000][1][gc,phases ] GC(363) Pre Evacuate Collection Set: 0.3ms

[2025-07-23T15:36:59.326+0000][1][gc,phases ] GC(363) Merge Heap Roots: 0.4ms

[2025-07-23T15:36:59.326+0000][1][gc,phases ] GC(363) Evacuate Collection Set: 80.0ms

[2025-07-23T15:36:59.326+0000][1][gc,phases ] GC(363) Post Evacuate Collection Set: 1.2ms

[2025-07-23T15:36:59.326+0000][1][gc,phases ] GC(363) Other: 0.6ms

[2025-07-23T15:36:59.326+0000][1][gc,heap ] GC(363) Eden regions: 208->0(115)

[2025-07-23T15:36:59.326+0000][1][gc,heap ] GC(363) Survivor regions: 300->85(85)

[2025-07-23T15:36:59.326+0000][1][gc,heap ] GC(363) Old regions: 2078->2496

[2025-07-23T15:36:59.326+0000][1][gc,heap ] GC(363) Archive regions: 2->2

[2025-07-23T15:36:59.326+0000][1][gc,heap ] GC(363) Humongous regions: 158->158

[2025-07-23T15:36:59.326+0000][1][gc,metaspace] GC(363) Metaspace: 93540K(96384K)->93540K(96384K) NonClass: 82360K(84404K)->82360K(84404K) Class: 11180K(11980K)->11180K(11980K)

[2025-07-23T15:36:59.326+0000][1][gc ] GC(363) Pause Young (Concurrent Start) (G1 Evacuation Pause) 10979M->10957M(14336M) 82.574ms

[2025-07-23T15:36:59.326+0000][1][gc,cpu ] GC(363) User=1.03s Sys=0.00s Real=0.08s

[2025-07-23T15:36:59.326+0000][1][gc ] GC(364) Concurrent Cycle

[2025-07-23T15:36:59.326+0000][1][gc,marking ] GC(364) Concurrent Clear Claimed Marks

[2025-07-23T15:36:59.326+0000][1][gc,marking ] GC(364) Concurrent Clear Claimed Marks 0.124ms

[2025-07-23T15:36:59.326+0000][1][gc,marking ] GC(364) Concurrent Scan Root Regions

[2025-07-23T15:36:59.399+0000][1][gc,marking ] GC(364) Concurrent Scan Root Regions 73.066ms

[2025-07-23T15:36:59.400+0000][1][gc,marking ] GC(364) Concurrent Mark (14728.882s)

[2025-07-23T15:36:59.400+0000][1][gc,marking ] GC(364) Concurrent Mark From Roots

[2025-07-23T15:36:59.400+0000][1][gc,task ] GC(364) Using 3 workers of 3 for marking

[2025-07-23T15:36:59.400+0000][1][gc,start ] GC(365) Pause Young (Normal) (G1 Evacuation Pause)

[2025-07-23T15:36:59.400+0000][1][gc,task ] GC(365) Using 13 workers of 13 for evacuation

[2025-07-23T15:36:59.400+0000][1][gc,age ] GC(365) Desired survivor size 71303168 bytes, new threshold 1 (max threshold 15)

[2025-07-23T15:36:59.431+0000][1][gc,age ] GC(365) Age table with threshold 1 (max threshold 15)

[2025-07-23T15:36:59.431+0000][1][gc,age ] GC(365) - age 1: 142342512 bytes, 142342512 total

[2025-07-23T15:36:59.432+0000][1][gc,phases ] GC(365) Pre Evacuate Collection Set: 52.7ms

[2025-07-23T15:36:59.432+0000][1][gc,phases ] GC(365) Merge Heap Roots: 0.1ms

[2025-07-23T15:36:59.432+0000][1][gc,phases ] GC(365) Evacuate Collection Set: 30.4ms

[2025-07-23T15:36:59.432+0000][1][gc,phases ] GC(365) Post Evacuate Collection Set: 1.0ms

[2025-07-23T15:36:59.432+0000][1][gc,phases ] GC(365) Other: 0.3ms

[2025-07-23T15:36:59.432+0000][1][gc,heap ] GC(365) Eden regions: 115->0(145)

[2025-07-23T15:36:59.432+0000][1][gc,heap ] GC(365) Survivor regions: 85->34(34)

[2025-07-23T15:36:59.432+0000][1][gc,heap ] GC(365) Old regions: 2496->2659

[2025-07-23T15:36:59.432+0000][1][gc,heap ] GC(365) Archive regions: 2->2

[2025-07-23T15:36:59.432+0000][1][gc,heap ] GC(365) Humongous regions: 159->159

[2025-07-23T15:36:59.432+0000][1][gc,metaspace] GC(365) Metaspace: 93540K(96384K)->93540K(96384K) NonClass: 82360K(84404K)->82360K(84404K) Class: 11180K(11980K)->11180K(11980K)

[2025-07-23T15:36:59.432+0000][1][gc ] GC(365) Pause Young (Normal) (G1 Evacuation Pause) 11421M->11409M(14336M) 32.083ms

[2025-07-23T15:36:59.432+0000][1][gc,cpu ] GC(365) User=0.39s Sys=0.00s Real=0.04s

[2025-07-23T15:36:59.455+0000][1][gc,start ] GC(366) Pause Young (Normal) (G1 Evacuation Pause)

[2025-07-23T15:36:59.455+0000][1][gc,task ] GC(366) Using 13 workers of 13 for evacuation

[2025-07-23T15:36:59.455+0000][1][gc,age ] GC(366) Desired survivor size 62914560 bytes, new threshold 1 (max threshold 15)

[2025-07-23T15:36:59.483+0000][1][gc,age ] GC(366) Age table with threshold 1 (max threshold 15)

[2025-07-23T15:36:59.483+0000][1][gc,age ] GC(366) - age 1: 125642440 bytes, 125642440 total

[2025-07-23T15:36:59.483+0000][1][gc,phases ] GC(366) Pre Evacuate Collection Set: 0.2ms

[2025-07-23T15:36:59.483+0000][1][gc,phases ] GC(366) Merge Heap Roots: 0.1ms

[2025-07-23T15:36:59.483+0000][1][gc,phases ] GC(366) Evacuate Collection Set: 26.5ms

[2025-07-23T15:36:59.483+0000][1][gc,phases ] GC(366) Post Evacuate Collection Set: 1.0ms

[2025-07-23T15:36:59.483+0000][1][gc,phases ] GC(366) Other: 0.3ms

[2025-07-23T15:36:59.483+0000][1][gc,heap ] GC(366) Eden regions: 145->0(149)

[2025-07-23T15:36:59.483+0000][1][gc,heap ] GC(366) Survivor regions: 34->30(30)

[2025-07-23T15:36:59.483+0000][1][gc,heap ] GC(366) Old regions: 2659->2801

[2025-07-23T15:36:59.483+0000][1][gc,heap ] GC(366) Archive regions: 2->2

[2025-07-23T15:36:59.483+0000][1][gc,heap ] GC(366) Humongous regions: 159->159

[2025-07-23T15:36:59.483+0000][1][gc,metaspace] GC(366) Metaspace: 93540K(96384K)->93540K(96384K) NonClass: 82360K(84404K)->82360K(84404K) Class: 11180K(11980K)->11180K(11980K)

[2025-07-23T15:36:59.483+0000][1][gc ] GC(366) Pause Young (Normal) (G1 Evacuation Pause) 11989M->11963M(14336M) 28.195ms

[2025-07-23T15:36:59.483+0000][1][gc,cpu ] GC(366) User=0.33s Sys=0.01s Real=0.03s

[DavidTurner]

@Christian_Dahlqvist

This version is very, very old and has been EOL a very long time. I would recommend that you upgrade to a supported version with urgency.

Seconded. This version is too old to be worth investigating further.

@VietBui

• what should I do next to investigate the root cause?

Upgrade to a supported version. If you can reproduce it in a supported version, you will need to look at the heap dump generated by the OOM event to understand its cause. If you need help with that then you can ask about it here. But it's very unlikely anyone will look at a heap dump from such an old version.

@VietBui

• Is there any workaround to prevent this problem?

There's a good chance that upgrading to a supported version will prevent it. If the problems continue in a supported version then we may be able to find a workaround by looking at the heap dump.

[VietBui]

Thank you for suggesting an upgrade to address our G1GC issues. However, our Elasticsearch cluster is in production, running in 3 data nodes in docker container, making upgrades challenging due to potential downtime and critical workloads (e.g., continuous indexing and queries). We’re seeking ways to diagnose and fix an OutOfMemory (OOM) risk without upgrading immediately, while planning a future upgrade in staging

[Christian_Dahlqvist]

In order to improve stability in the version you are running I would try switching to CMS GC, which I believe was the default for that version. As David mentioned it may be difficult to find someone willing to help spend time troubleshooting this for such an old version that is so far past EOL and no longer in common use.

[DavidTurner]

@VietBui

upgrades challenging due to potential downtime and critical workloads (e.g., continuous indexing and queries)

Elasticsearch supports rolling upgrades, so you can keep on doing indexing and queries while doing the upgrade.

[RainTown (Kevin Maguire)]

Ages ago I asked, amongst other questions:

@RainTown

Are the "To-space exhausted" message new/recent, or were they always there?

My understanding from your response was that this was “new”, including actual OOM-ing, though admittedly you were not completely explicit.

There would be a point, in terms of load/size/usage/whatever, when every solution starts to bend. In ideal circumstances, you are able to anticipate this and .. take some action. If your production solution has been fine for ages, on 6.8.22 with G1GC, then not fine on exactly same version/GC, then maybe you reached that “starting to bend” point? All things being equal, these things are fairly deterministic, so if there’s a change in observed behaviour then highly likely something else changed too.

Changing to CMS might help, might also make it worse. If you have no other options, it’s worth trying.

If you really really can’t change anything, then dig into the weeds, ie details of precisely what is using your heap, using JVM troubleshooting tools, and try work that back to your actual workload/queries/usage patterns/etc. but be aware that this can be pretty hard and could easily lead nowhere useful.

In passing:

@VietBui

our Elasticsearch cluster is in production, running in 3 data nodes in docker container, making upgrades challenging due to potential downtime and critical workloads (e.g., continuous indexing and queries).

That’s unfortunately (probably) a big part of why you are where you are. Kick the can today, carry the burden tomorrow.

Exec summary - you seem to be looking for a magic bullet. There’s a lot of bright people on this forum, none of them have yet found it. Work on premise that it might not exist!