# Elastic Cluster Red - How to fix! **URL:** \ **Category:** Elasticsearch\ **Created:** [May 15, 2019, 7:42am UTC](https://discuss.elastic.co/t/elastic-cluster-red-how-to-fix/181114 "2019-05-15T07:42:04Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\ **Post date:** [May 15, 2019, 7:42am UTC](https://discuss.elastic.co/t/elastic-cluster-red-how-to-fix/181114/1 "2019-05-15T07:42:04Z")
Hi all, I have a Elastic Cluster with 6 nodes in 6 different hosts. The elastic cluster status is red. I am using the latest elasticsearch version 7.0.1 to all the nodes. GET \_cluster/health { "cluster\_name" : "xh-elastic-cluster-1", "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 6, "number\_of\_data\_nodes" : 6, "active\_primary\_shards" : 547, "active\_shards" : 1028, "relocating\_shards" : 0, "initializing\_shards" : 3, "unassigned\_shards" : 67, "delayed\_unassigned\_shards" : 0, "number\_of\_pending\_tasks" : 0, "number\_of\_in\_flight\_fetch" : 0, "task\_max\_waiting\_in\_queue\_millis" : 0, "active\_shards\_percent\_as\_number" : 93.6247723132969 } On Monitoring on the Cluster i also see Unassigned Shards: 16 What i have to do to fix the cluster status? ![1](https://us1.discourse-cdn.com/elastic/original/3X/c/7/c79fc3aeb8692eedc51cb1ca0a57b5144b5e4f4c.png) Best Regards, Thanos ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 15, 2019, 11:22am UTC](https://discuss.elastic.co/t/elastic-cluster-red-how-to-fix/181114/2 "2019-05-15T11:22:36Z")
Here's a blog post that should help you work out what's going wrong: > **[RED Elasticsearch Cluster? Panic no longer](https://www.elastic.co/blog/red-elasticsearch-cluster-panic-no-longer)** > > Your cluster is RED - you have unassigned shards and your cluster is not fully operational. Introducing a new Elasticsearch API which will save the day. ---
**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\ **Post date:** [May 15, 2019, 12:34pm UTC](https://discuss.elastic.co/t/elastic-cluster-red-how-to-fix/181114/3 "2019-05-15T12:34:40Z")
Thank you very much for this Blog. I run the GET /\_cluster/allocation/explain and i got ```auto { "index" : "metricbeat-6.7.0-2019.05.13", "shard" : 0, "primary" : true, "current_state" : "unassigned", "unassigned_info" : { "reason" : "MANUAL_ALLOCATION", "at" : "2019-05-13T08:14:06.344Z", "details" : "failed shard on node [wIwQPrBRRuKRxR9_OtJxhw]: shard failure, reason [failed to recover from translog], failure EngineException[failed to recover from translog]; nested: IllegalIndexShardStateException[CurrentState[CLOSED] operation only allowed when recovering, origin [LOCAL_TRANSLOG_RECOVERY]]; ", "last_allocation_status" : "no_valid_shard_copy" }, "can_allocate" : "no_valid_shard_copy", "allocate_explanation" : "cannot allocate because all found copies of the shard are either stale or corrupt"...... ``` What can i do in such a case? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 15, 2019, 4:10pm UTC](https://discuss.elastic.co/t/elastic-cluster-red-how-to-fix/181114/4 "2019-05-15T16:10:06Z")
Can you share the whole output? ---
**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\ **Post date:** [May 16, 2019, 7:29am UTC](https://discuss.elastic.co/t/elastic-cluster-red-how-to-fix/181114/5 "2019-05-16T07:29:25Z")
> [@antonopo](#): > > GET /\_cluster/allocation/explain Hi, Yesterday i found the indices with the 0 docs count and i deleted them. After this action the status of the Cluster became yellow. Today at the morning was red again and i had to delete a red status index. After that the cluster status is: GET \_cluster/health { "cluster\_name" : "xh-elastic-cluster-1", "status" : "yellow", "timed\_out" : false, "number\_of\_nodes" : 6, "number\_of\_data\_nodes" : 6, "active\_primary\_shards" : 520, "active\_shards" : 995, "relocating\_shards" : 2, "initializing\_shards" : 0, "unassigned\_shards" : 45, "delayed\_unassigned\_shards" : 0, "number\_of\_pending\_tasks" : 0, "number\_of\_in\_flight\_fetch" : 0, "task\_max\_waiting\_in\_queue\_millis" : 0, "active\_shards\_percent\_as\_number" : 95.67307692307693 } GET /\_cluster/allocation/explain { "index" : "winlogbeat-6.7.0-2019.05.03", "shard" : 1, "primary" : false, "current\_state" : "unassigned", "unassigned\_info" : { "reason" : "NODE\_LEFT", "at" : "2019-05-16T07:16:59.621Z", "details" : "node\_left [6q5asfwjQ\_eoI3xkl2-JXg]", "last\_allocation\_status" : "no\_attempt" }, "can\_allocate" : "throttled", "allocate\_explanation" : "allocation temporarily throttled", "node\_allocation\_decisions" : [ { "node\_id" : "14b6D9VCR36schkKD3k74A", "node\_name" : "xh-fr-elastic-2", "transport\_address" : "135.238.239.132:9300", "node\_attributes" : { "ml.machine\_memory" : "269930725376", "ml.max\_open\_jobs" : "20", "xpack.installed" : "true" }, "node\_decision" : "throttled", "deciders" : [ { "decider" : "throttling", "decision" : "THROTTLE", "explanation" : "reached the limit of outgoing shard recoveries [2] on the node [isTX9Dk7SMSaP3GARPtU9A] which holds the primary, cluster setting [cluster.routing.allocation.node\_concurrent\_outgoing\_recoveries=2] (can also be set via [cluster.routing.allocation.node\_concurrent\_recoveries])" } ] }, { "node\_id" : "6q5asfwjQ\_eoI3xkl2-JXg", "node\_name" : "xh-gr-elastic-1", "transport\_address" : "10.158.67.175:9300", "node\_attributes" : { "ml.machine\_memory" : "16654872576", "ml.max\_open\_jobs" : "20", "xpack.installed" : "true" }, "node\_decision" : "throttled", "deciders" : [ { "decider" : "throttling", "decision" : "THROTTLE", "explanation" : "reached the limit of outgoing shard recoveries [2] on the node [isTX9Dk7SMSaP3GARPtU9A] which holds the primary, cluster setting [cluster.routing.allocation.node\_concurrent\_outgoing\_recoveries=2] (can also be set via [cluster.routing.allocation.node\_concurrent\_recoveries])" } ] }, { "node\_id" : "OETgEHqTR9Ku30WwPWADyg", "node\_name" : "xh-gr-elastic-2", "transport\_address" : "10.159.166.9:9300", "node\_attributes" : { "ml.machine\_memory" : "269932404736", "ml.max\_open\_jobs" : "20", "xpack.installed" : "true" }, "node\_decision" : "throttled", "deciders" : [ { "decider" : "throttling", "decision" : "THROTTLE", "explanation" : "reached the limit of outgoing shard recoveries [2] on the node [isTX9Dk7SMSaP3GARPtU9A] which holds the primary, cluster setting [cluster.routing.allocation.node\_concurrent\_outgoing\_recoveries=2] (can also be set via [cluster.routing.allocation.node\_concurrent\_recoveries])" } ] }, { "node\_id" : "\_BIH8swLQz6rJa\_ImpYJuA", "node\_name" : "xh-it-elastic-2", "transport\_address" : "151.98.17.34:9300", "node\_attributes" : { "ml.machine\_memory" : "8186564608", "ml.max\_open\_jobs" : "20", "xpack.installed" : "true" }, "node\_decision" : "throttled", "deciders" : [ { "decider" : "throttling", "decision" : "THROTTLE", "explanation" : "reached the limit of incoming shard recoveries [2], cluster setting [cluster.routing.allocation.node\_concurrent\_incoming\_recoveries=2] (can also be set via [cluster.routing.allocation.node\_concurrent\_recoveries])" } ] }, { "node\_id" : "uHTHPU56QfmRVfP29OsS0Q", "node\_name" : "xh-it-elastic-1", "transport\_address" : "151.98.17.60:9300", "node\_attributes" : { "ml.machine\_memory" : "9223372036854771712", "ml.max\_open\_jobs" : "20", "xpack.installed" : "true" }, "node\_decision" : "throttled", "deciders" : [ { "decider" : "throttling", "decision" : "THROTTLE", "explanation" : "reached the limit of outgoing shard recoveries [2] on the node [isTX9Dk7SMSaP3GARPtU9A] which holds the primary, cluster setting [cluster.routing.allocation.node\_concurrent\_outgoing\_recoveries=2] (can also be set via [cluster.routing.allocation.node\_concurrent\_recoveries])" } ] }, { "node\_id" : "isTX9Dk7SMSaP3GARPtU9A", "node\_name" : "xh-fr-elastic-1", "transport\_address" : "135.238.239.48:9300", "node\_attributes" : { "ml.machine\_memory" : "16654970880", "ml.max\_open\_jobs" : "20", "xpack.installed" : "true" }, "node\_decision" : "no", "deciders" : [ { "decider" : "same\_shard", "decision" : "NO", "explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[winlogbeat-6.7.0-2019.05.03][1], node[isTX9Dk7SMSaP3GARPtU9A], [P], s[STARTED], a[id=N2QSmunAQEGqKED\_jnKKBA]]" }, { "decider" : "throttling", "decision" : "THROTTLE", "explanation" : "reached the limit of outgoing shard recoveries [2] on the node [isTX9Dk7SMSaP3GARPtU9A] which holds the primary, cluster setting [cluster.routing.allocation.node\_concurrent\_outgoing\_recoveries=2] (can also be set via [cluster.routing.allocation.node\_concurrent\_recoveries])" } ] } ] } ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 16, 2019, 8:23am UTC](https://discuss.elastic.co/t/elastic-cluster-red-how-to-fix/181114/6 "2019-05-16T08:23:21Z")
> [@antonopo](#): > > Today at the morning was red again and i had to delete a red status index. After that the cluster status is: This doesn't tell us anything about why your cluster health was red. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 13, 2019, 8:23am UTC](https://discuss.elastic.co/t/elastic-cluster-red-how-to-fix/181114/7 "2019-06-13T08:23:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.