# Shards getting marked as stale frequently causing cluster to go yellow

**URL:** <https://discuss.elastic.co/t/shards-getting-marked-as-stale-frequently-causing-cluster-to-go-yellow/231835>\
**Category:** Elasticsearch\
**Created:** [May 9, 2020, 11:06am UTC](https://discuss.elastic.co/t/shards-getting-marked-as-stale-frequently-causing-cluster-to-go-yellow/231835 "2020-05-09T11:06:03Z")\
**Posts on this page:** 6\
**Page:** 2

<div class="post-metadata">

**Author:** ![Faiz\_Ahmed\_Mushtak\_H](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/faiz_ahmed_mushtak_h/32/47387_2.png) [@Faiz\_Ahmed\_Mushtak\_H](https://discuss.elastic.co/u/Faiz_Ahmed_Mushtak_H)\
**Post date:** [May 11, 2020, 7:23pm UTC](https://discuss.elastic.co/t/shards-getting-marked-as-stale-frequently-causing-cluster-to-go-yellow/231835/21 "2020-05-11T19:23:09Z")

</div>

I meant its array of strings (with duplicates). Eventually a string will be there in memory when the JSON is parsed right (for the ctx.\_source), be it the document or the request that is sent. I was talking about that

And yes, even the JSON object could grow too

---

<div class="post-metadata">

**Author:** ![Faiz\_Ahmed\_Mushtak\_H](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/faiz_ahmed_mushtak_h/32/47387_2.png) [@Faiz\_Ahmed\_Mushtak\_H](https://discuss.elastic.co/u/Faiz_Ahmed_Mushtak_H)\
**Post date:** [May 13, 2020, 7:12pm UTC](https://discuss.elastic.co/t/shards-getting-marked-as-stale-frequently-causing-cluster-to-go-yellow/231835/22 "2020-05-13T19:12:57Z")

</div>

Reducing the JSON size seems to have done the trick. We'll observe for a few more days. But before that, can you tell me what's the impact of increasing g1heapregionsize? What should I expect if I adjust it from 8mb to 16mb? I'll go thru our GC logs and see how many humongous regions are being created and compare it with previous incidents to verify if we should even tweak region size

---

<div class="post-metadata">

**Author:** ![Faiz\_Ahmed\_Mushtak\_H](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/faiz_ahmed_mushtak_h/32/47387_2.png) [@Faiz\_Ahmed\_Mushtak\_H](https://discuss.elastic.co/u/Faiz_Ahmed_Mushtak_H)\
**Post date:** [May 16, 2020, 11:01am UTC](https://discuss.elastic.co/t/shards-getting-marked-as-stale-frequently-causing-cluster-to-go-yellow/231835/23 "2020-05-16T11:01:52Z")

</div>

Almost a week without circuit breaker tripping. Looks like it has helped!

Just a minor concern, do you happen to know how g1gc behaves when snapshots are taken? We see a spike in our young GC time (goes upto 500ms-1s). We also see number of young GC to increase. So maybe the spike in time is a consequence of frequency of GC

Elasticsearch however wasn't affected in any noticeable way. The way young gen time is provided in the node stats (via JVM API's most likely), does it take into consideration time taken for concurrent phases, or only paused phases? We also have seen an increase in time taken to dedupe strings when snapshots are being taken. My guess is G1GC is taken by surprise because of the memory pressure the snapshot puts onto the heap

Any recommendations to reduce young gen frequency

More info available here

> <https://stackoverflow.com/questions/61835474/g1gc-surge-in-number-of-young-collection>

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 16, 2020, 12:16pm UTC](https://discuss.elastic.co/t/shards-getting-marked-as-stale-frequently-causing-cluster-to-go-yellow/231835/24 "2020-05-16T12:16:31Z")

</div>

It's certainly possible: taking a snapshot does involve moving a load of data around and maybe we're creating more garbage than needed. There's been a lot of work on streamlining the snapshot process since 7.2. Off the top of my head I'm not sure if any of it would directly affect memory pressure but it might. Can you reproduce this in 7.7?

---

<div class="post-metadata">

**Author:** ![Faiz\_Ahmed\_Mushtak\_H](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/faiz_ahmed_mushtak_h/32/47387_2.png) [@Faiz\_Ahmed\_Mushtak\_H](https://discuss.elastic.co/u/Faiz_Ahmed_Mushtak_H)\
**Post date:** [May 16, 2020, 12:24pm UTC](https://discuss.elastic.co/t/shards-getting-marked-as-stale-frequently-causing-cluster-to-go-yellow/231835/25 "2020-05-16T12:24:44Z")

</div>

I'll need to check that @DavidTurner  
We may plan for an in-place upgrade as elasticsearch has already reached 7.7

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 13, 2020, 12:24pm UTC](https://discuss.elastic.co/t/shards-getting-marked-as-stale-frequently-causing-cluster-to-go-yellow/231835/26 "2020-06-13T12:24:56Z")

</div>

Closed on June 13, 2020, 12:24pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/shards-getting-marked-as-stale-frequently-causing-cluster-to-go-yellow/231835.md?page=1)
