# Index Failover to Warm node not happening

**URL:** <https://discuss.elastic.co/t/index-failover-to-warm-node-not-happening/248199>\
**Category:** Elasticsearch\
**Tags:** ilm-index-lifecycle-management\
**Created:** [September 10, 2020, 4:04pm UTC](https://discuss.elastic.co/t/index-failover-to-warm-node-not-happening/248199 "2020-09-10T16:04:04Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ajesh](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@ajesh](https://discuss.elastic.co/u/ajesh)\
**Post date:** [September 10, 2020, 4:04pm UTC](https://discuss.elastic.co/t/index-failover-to-warm-node-not-happening/248199/1 "2020-09-10T16:04:04Z")

</div>

Hi Team,

My index is not rolling over to Warm nodes as per the ILM policy , it just simply grows. Please see the attached . ILM policy is set to Index size 50GB or 30 Days. Index stats and screenshots attached , i could not find any issues

Please help

 ![index_sc](https://us1.discourse-cdn.com/elastic/original/3X/2/8/2802e087b66019d577276be138d8e4711b2cc4fa.jpeg) ![index template](https://us1.discourse-cdn.com/elastic/original/3X/d/0/d05bf5fe79502969e1f94bc984a388730973705a.jpeg) ![ILM2](https://us1.discourse-cdn.com/elastic/original/3X/a/d/adcbc52c99770d56b9778dc9c01c5a7112a5ceb2.jpeg) ![ILM1](https://us1.discourse-cdn.com/elastic/original/3X/b/b/bbac1da9f24a1b4142c79268a610e727c126fbe0.jpeg)

> {  
> "\_shards": {  
> "total": 2,  
> "successful": 2,  
> "failed": 0  
> },  
> "stats": {  
> "uuid": "g6NKgG6VT8ucP80NChpCxQ",  
> "primaries": {  
> "docs": {  
> "count": 58067343,  
> "deleted": 0  
> },  
> "store": {  
> "size\_in\_bytes": 46527647876,  
> "reserved\_in\_bytes": 0  
> },  
> "indexing": {  
> "index\_total": 2153862,  
> "index\_time\_in\_millis": 930895,  
> "index\_current": 0,  
> "index\_failed": 0,  
> "delete\_total": 0,  
> "delete\_time\_in\_millis": 0,  
> "delete\_current": 0,  
> "noop\_update\_total": 0,  
> "is\_throttled": false,  
> "throttle\_time\_in\_millis": 0  
> },  
> "get": {  
> "total": 0,  
> "time\_in\_millis": 0,  
> "exists\_total": 0,  
> "exists\_time\_in\_millis": 0,  
> "missing\_total": 0,  
> "missing\_time\_in\_millis": 0,  
> "current": 0  
> },  
> "search": {  
> "open\_contexts": 0,  
> "query\_total": 13,  
> "query\_time\_in\_millis": 868,  
> "query\_current": 0,  
> "fetch\_total": 13,  
> "fetch\_time\_in\_millis": 0,  
> "fetch\_current": 0,  
> "scroll\_total": 0,  
> "scroll\_time\_in\_millis": 0,  
> "scroll\_current": 0,  
> "suggest\_total": 0,  
> "suggest\_time\_in\_millis": 0,  
> "suggest\_current": 0  
> },  
> "merges": {  
> "current": 2,  
> "current\_docs": 7361664,  
> "current\_size\_in\_bytes": 5884551813,  
> "total": 50,  
> "total\_time\_in\_millis": 662880,  
> "total\_docs": 4826037,  
> "total\_size\_in\_bytes": 4741719006,  
> "total\_stopped\_time\_in\_millis": 18663,  
> "total\_throttled\_time\_in\_millis": 360156,  
> "total\_auto\_throttle\_in\_bytes": 10584561  
> },  
> "refresh": {  
> "total": 111,  
> "total\_time\_in\_millis": 57264,  
> "external\_total": 98,  
> "external\_total\_time\_in\_millis": 56530,  
> "listeners": 0  
> },  
> "flush": {  
> "total": 9,  
> "periodic": 9,  
> "total\_time\_in\_millis": 10576  
> },  
> "warmer": {  
> "current": 0,  
> "total": 97,  
> "total\_time\_in\_millis": 12  
> },  
> "query\_cache": {  
> "memory\_size\_in\_bytes": 0,  
> "total\_count": 1312,  
> "hit\_count": 0,  
> "miss\_count": 1312,  
> "cache\_size": 0,  
> "cache\_count": 0,  
> "evictions": 0  
> },  
> "fielddata": {  
> "memory\_size\_in\_bytes": 0,  
> "evictions": 0  
> },  
> "completion": {  
> "size\_in\_bytes": 0  
> },  
> "segments": {  
> "count": 56,  
> "memory\_in\_bytes": 2781120,  
> "terms\_memory\_in\_bytes": 1718304,  
> "stored\_fields\_memory\_in\_bytes": 528336,  
> "term\_vectors\_memory\_in\_bytes": 0,  
> "norms\_memory\_in\_bytes": 0,  
> "points\_memory\_in\_bytes": 0,  
> "doc\_values\_memory\_in\_bytes": 534480,  
> "index\_writer\_memory\_in\_bytes": 23996252,  
> "version\_map\_memory\_in\_bytes": 0,  
> "fixed\_bit\_set\_memory\_in\_bytes": 0,  
> "max\_unsafe\_auto\_id\_timestamp": 1599753387064,  
> "file\_sizes": {}  
> },  
> "translog": {  
> "operations": 251207,  
> "size\_in\_bytes": 534692472,  
> "uncommitted\_operations": 251207,  
> "uncommitted\_size\_in\_bytes": 534692472,  
> "earliest\_last\_modified\_age": 0  
> },  
> "request\_cache": {  
> "memory\_size\_in\_bytes": 0,  
> "evictions": 0,  
> "hit\_count": 0,  
> "miss\_count": 0  
> },  
> "recovery": {  
> "current\_as\_source": 0,  
> "current\_as\_target": 0,  
> "throttle\_time\_in\_millis": 0  
> }  
> },  
> "total": {  
> "docs": {  
> "count": 116133741,  
> "deleted": 0  
> },  
> "store": {  
> "size\_in\_bytes": 92558575289,  
> "reserved\_in\_bytes": 0  
> },  
> "indexing": {  
> "index\_total": 4307717,  
> "index\_time\_in\_millis": 1839373,  
> "index\_current": 1,  
> "index\_failed": 0,  
> "delete\_total": 0,  
> "delete\_time\_in\_millis": 0,  
> "delete\_current": 0,  
> "noop\_update\_total": 0,  
> "is\_throttled": false,  
> "throttle\_time\_in\_millis": 0  
> },  
> "get": {  
> "total": 0,  
> "time\_in\_millis": 0,  
> "exists\_total": 0,  
> "exists\_time\_in\_millis": 0,  
> "missing\_total": 0,  
> "missing\_time\_in\_millis": 0,  
> "current": 0  
> },  
> "search": {  
> "open\_contexts": 0,  
> "query\_total": 20,  
> "query\_time\_in\_millis": 2074,  
> "query\_current": 0,  
> "fetch\_total": 20,  
> "fetch\_time\_in\_millis": 0,  
> "fetch\_current": 0,  
> "scroll\_total": 0,  
> "scroll\_time\_in\_millis": 0,  
> "scroll\_current": 0,  
> "suggest\_total": 0,  
> "suggest\_time\_in\_millis": 0,  
> "suggest\_current": 0  
> },  
> "merges": {  
> "current": 4,  
> "current\_docs": 13496833,  
> "current\_size\_in\_bytes": 11315727435,  
> "total": 98,  
> "total\_time\_in\_millis": 1332893,  
> "total\_docs": 9255286,  
> "total\_size\_in\_bytes": 8925772873,  
> "total\_stopped\_time\_in\_millis": 54120,  
> "total\_throttled\_time\_in\_millis": 739319,  
> "total\_auto\_throttle\_in\_bytes": 23286034  
> },  
> "refresh": {  
> "total": 220,  
> "total\_time\_in\_millis": 115113,  
> "external\_total": 196,  
> "external\_total\_time\_in\_millis": 114757,  
> "listeners": 0  
> },  
> "flush": {  
> "total": 19,  
> "periodic": 18,  
> "total\_time\_in\_millis": 15781  
> },  
> "warmer": {  
> "current": 0,  
> "total": 194,  
> "total\_time\_in\_millis": 26  
> },  
> "query\_cache": {  
> "memory\_size\_in\_bytes": 0,  
> "total\_count": 1959,  
> "hit\_count": 0,  
> "miss\_count": 1959,  
> "cache\_size": 0,  
> "cache\_count": 0,  
> "evictions": 0  
> },  
> "fielddata": {  
> "memory\_size\_in\_bytes": 0,  
> "evictions": 0  
> },  
> "completion": {  
> "size\_in\_bytes": 0  
> },  
> "segments": {  
> "count": 109,  
> "memory\_in\_bytes": 5412300,  
> "terms\_memory\_in\_bytes": 3374816,  
> "stored\_fields\_memory\_in\_bytes": 1054936,  
> "term\_vectors\_memory\_in\_bytes": 0,  
> "norms\_memory\_in\_bytes": 0,  
> "points\_memory\_in\_bytes": 0,  
> "doc\_values\_memory\_in\_bytes": 982548,  
> "index\_writer\_memory\_in\_bytes": 49080332,  
> "version\_map\_memory\_in\_bytes": 0,  
> "fixed\_bit\_set\_memory\_in\_bytes": 0,  
> "max\_unsafe\_auto\_id\_timestamp": 1599753387064,  
> "file\_sizes": {}  
> },  
> "translog": {  
> "operations": 434737,  
> "size\_in\_bytes": 925679915,  
> "uncommitted\_operations": 434737,  
> "uncommitted\_size\_in\_bytes": 925679915,  
> "earliest\_last\_modified\_age": 0  
> },  
> "request\_cache": {  
> "memory\_size\_in\_bytes": 0,  
> "evictions": 0,  
> "hit\_count": 0,  
> "miss\_count": 0  
> },  
> "recovery": {  
> "current\_as\_source": 0,  
> "current\_as\_target": 0,  
> "throttle\_time\_in\_millis": 0  
> }  
> }  
> }  
> }

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 12, 2020, 9:02am UTC](https://discuss.elastic.co/t/index-failover-to-warm-node-not-happening/248199/2 "2020-09-12T09:02:06Z")

</div>

The specified size refers to primary shard size. Does the size you are looking at by any chance include the size of replica shards as well?

---

<div class="post-metadata">

**Author:** ![ajesh](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@ajesh](https://discuss.elastic.co/u/ajesh)\
**Post date:** [September 12, 2020, 4:56pm UTC](https://discuss.elastic.co/t/index-failover-to-warm-node-not-happening/248199/3 "2020-09-12T16:56:19Z")

</div>

Hi Christian,

My understanding was it will failover when total size exceeds (including primary and replica shard) the set size.

failover happed to warm node when it crossed 100GB.

Thank you Christian

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 12, 2020, 5:07pm UTC](https://discuss.elastic.co/t/index-failover-to-warm-node-not-happening/248199/4 "2020-09-12T17:07:28Z")

</div>

Given that you can change the number of replica shards at any time the behavior could be very unpredictable if full size was considered.

---

<div class="post-metadata">

**Author:** ![ajesh](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@ajesh](https://discuss.elastic.co/u/ajesh)\
**Post date:** [September 13, 2020, 3:45pm UTC](https://discuss.elastic.co/t/index-failover-to-warm-node-not-happening/248199/5 "2020-09-13T15:45:13Z")

</div>

Thanks again for your help and support.

we are currently going with replica: 1 for all indices as of now and planning to archive to s3.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 11, 2020, 3:45pm UTC](https://discuss.elastic.co/t/index-failover-to-warm-node-not-happening/248199/6 "2020-10-11T15:45:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.