# Shards randomly going into "initializing"/"unassigned" state even when the cluster is in stable state **URL:** \ **Category:** Elasticsearch\ **Created:** [April 6, 2021, 12:34pm UTC](https://discuss.elastic.co/t/shards-randomly-going-into-initializing-unassigned-state-even-when-the-cluster-is-in-stable-state/269329 "2021-04-06T12:34:22Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![dinesh\_gnanasamy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dinesh_gnanasamy/32/63402_2.png) [@dinesh\_gnanasamy](https://discuss.elastic.co/u/dinesh_gnanasamy)\ **Post date:** [April 6, 2021, 12:34pm UTC](https://discuss.elastic.co/t/shards-randomly-going-into-initializing-unassigned-state-even-when-the-cluster-is-in-stable-state/269329/1 "2021-04-06T12:34:22Z")
**Description** : Even without any nodal disturbances (nodes leaving/joining), some shards go into INITIALIZING, UNASSIGNED states randomly over 24-48 hours. **Environment** : _ES Version_ : 7.0.1 _No of nodes_ : 12 (No specific roles - all nodes assume all roles) _Are indices static ?_ : 50% static indices, 50% rollover indices with rollover period starting from 2 hrs minimum _Deployment_ : All 12 ES nodes run as Kubernetes PODs _Metrics Collection_ : We use [telegraf](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/elasticsearch) to collect metrics and store the same in InfluxDB for ES-specific metrics tracking **Index-Specific Configurations** : 1. 10 shards per index with 1 replica each **Observations so far** : - Heap, Disk and Ram usage of the instances do not exceed anything more than 60% (from node stats) - No abnormal master elections seen - Kubernetes liveness probes do not fail (hence we can eliminate possibility of any node not responding to Kiubernetes and hence gets restarted by Kubernetes) - No nodal restarts - No abnormal load on master node - Checked if rollover of indices (destroy old indices and creating new indices) cause state-change in the shards - Not all rollover indices go into YELLOW state, less than 5% of total count of rollover indices get into YELLOW state. Sometimes they remain in YELLOW state after giving up all allocation-attempts - but even this pattern is not consistent - in few instances they also recover neatly and move to GREEN state over time **Clarifications** : 1. While I understand that `_cluster/allocation/explain?pretty"` can provide insight on reason for `UNASSIGNED` state, my understanding was that any shard-state change is only triggered by nodal-state changes (joining / leaving etc.) - which is not the case here. Hence, what are other reasons which could trigger changes in shard-states ? 2. Checked if there are any abnormal logs on master - found none. If you could point to anything specific to look into master and/or other nodes, please let me know 3. Any specific cluster / nodal stats to be examined to correlate with this behaviour ? 4. _More importantly_, will replica-shards for fewer indices remaining in `UNASSIGNED` state impact queries on those indices ? As I understand , searches use Adaptive Replica Selection. and some replicas are `UNASSIGNED`, I assume that in such scenario, queries would hammer primary-shards more vigorously (or am I over-exaggerating?) Pls correct me if I am wrong. If my assumption is right, please let me know of any search-metrics to correlate with ? I can share the `_cluster/allocation/explain?pretty"` output, but wanted to understand what reasons other than nodal joins/leaves can cause the change in shard-states. Thanks in advance - Dinesh ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 6, 2021, 5:26pm UTC](https://discuss.elastic.co/t/shards-randomly-going-into-initializing-unassigned-state-even-when-the-cluster-is-in-stable-state/269329/2 "2021-04-06T17:26:33Z")
> [@dinesh\_gnanasamy](#): > > _ES Version_ : 7.0.1 That's pretty old, long past [EOL](https://www.elastic.co/support/eol), so you should upgrade as a matter of urgency. I don't think I have a development environment for this version any more so detailed troubleshooting will be difficult. > [@dinesh\_gnanasamy](#): > > Checked if there are any abnormal logs on master - found none. If you could point to anything specific to look into master and/or other nodes, please let me know I'm pretty sure all shard failures are logged, and the message will indicate the reason. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 4, 2021, 5:27pm UTC](https://discuss.elastic.co/t/shards-randomly-going-into-initializing-unassigned-state-even-when-the-cluster-is-in-stable-state/269329/3 "2021-05-04T17:27:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.