# Health check on my ES cluster shows 99.7% **URL:** \ **Category:** Elasticsearch\ **Created:** [February 27, 2019, 12:49am UTC](https://discuss.elastic.co/t/health-check-on-my-es-cluster-shows-99-7/170082 "2019-02-27T00:49:51Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![drivera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drivera/32/39900_2.png) [@drivera](https://discuss.elastic.co/u/drivera)\ **Post date:** [February 27, 2019, 12:49am UTC](https://discuss.elastic.co/t/health-check-on-my-es-cluster-shows-99-7/170082/1 "2019-02-27T00:49:51Z")
After unsuccessfully trying to backup my cluster, I bounced all my nodes and now I see the error below. Any ideas? Events are still coming but I'm concerned. 1551228466 16:47:46 istunixes yellow 9 5 2087 1047 0 0 7 0 - 99.7% ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 27, 2019, 9:33am UTC](https://discuss.elastic.co/t/health-check-on-my-es-cluster-shows-99-7/170082/2 "2019-02-27T09:33:24Z")
Is the cluster still in this state? Are there ongoing recoveries? When you restart nodes it can take some time to recover all the shards. Check the [cluster allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html) if there are no ongoing recoveries and yet some shards are not allocated. ---
**Author:** ![drivera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drivera/32/39900_2.png) [@drivera](https://discuss.elastic.co/u/drivera)\ **Post date:** [February 27, 2019, 6:07pm UTC](https://discuss.elastic.co/t/health-check-on-my-es-cluster-shows-99-7/170082/3 "2019-02-27T18:07:42Z")
Hi David, the cluster is still in the same state. Read over the docs in the link you sent but I don't see the API response for my condition. I have included it below, sounds it may be an easy fix, in the end if I have to wipe that index, that is okay too. See the output of the GET /\_cluster/allocation/explain API: { "index": "syslog-rsyslog-2019-02-27", "shard": 0, "primary": true, "current\_state": "started", "current\_node": { "id": "AAtmertMTeiuE\_i5CQTKAA", "name": "ist000248", "transport\_address": "10.44.0.180:9300", "weight\_ranking": 4 }, "can\_remain\_on\_current\_node": "yes", "can\_rebalance\_cluster": "no", "can\_rebalance\_cluster\_decisions": [ { "decider": "cluster\_rebalance", "decision": "NO", "explanation": "the cluster has unassigned shards and cluster setting [cluster.routing.allocation.allow\_rebalance] is set to [indices\_all\_active]" } ], "can\_rebalance\_to\_other\_node": "no", "rebalance\_explanation": "rebalancing is not allowed, even though there is at least one node on which the shard can be allocated", "node\_allocation\_decisions": [ { "node\_id": "YQXjt41cQrK7vDHuNXubZA", "node\_name": "ist000242", "transport\_address": "10.44.3.8:9300", "node\_decision": "yes", "weight\_ranking": 1 }, { "node\_id": "ihHNmtfqTGSugWowHxCILw", "node\_name": "ist000243", "transport\_address": "10.44.2.206:9300", "node\_decision": "yes", "weight\_ranking": 2 }, { "node\_id": "H8dbQ6dnQtSIOadYeqGz6w", "node\_name": "ist000245", "transport\_address": "10.44.0.157:9300", "node\_decision": "yes", "weight\_ranking": 3 }, { "node\_id": "fCfm\_WQXTgeG52ca37Th1A", "node\_name": "ist000247", "transport\_address": "10.44.0.188:9300", "node\_decision": "no", "weight\_ranking": 4, "deciders": [ { "decider": "same\_shard", "decision": "NO", "explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[syslog-rsyslog-2019-02-27][0], node[fCfm\_WQXTgeG52ca37Th1A], [R], s[STARTED], a[id=p4ECJ365RV2gAzmvJWbp6g]]" } ] ---
**Author:** ![drivera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drivera/32/39900_2.png) [@drivera](https://discuss.elastic.co/u/drivera)\ **Post date:** [February 27, 2019, 6:19pm UTC](https://discuss.elastic.co/t/health-check-on-my-es-cluster-shows-99-7/170082/4 "2019-02-27T18:19:41Z")
Posting my current cluster settings: { "persistent": { "cluster": { "routing": { "allocation": { "cluster\_concurrent\_rebalance": "4", "node\_concurrent\_recoveries": "4", "disk": { "watermark": { "low": "90%", "high": "95%" } }, "enable": "all" } } }, "logger": { "\_root": "INFO" } }, "transient": {} } ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 28, 2019, 9:08am UTC](https://discuss.elastic.co/t/health-check-on-my-es-cluster-shows-99-7/170082/5 "2019-02-28T09:08:39Z")
> [@drivera](#): > > See the output of the GET /\_cluster/allocation/explain API: > > ```auto > { > "index": "syslog-rsyslog-2019-02-27", > "shard": 0, > "primary": true, > "current_state": "started", > "current_node": { > "id": "AAtmertMTeiuE_i5CQTKAA", > "name": "ist000248", > "transport_address": "10.44.0.180:9300", > > ``` Hmm, that's strange, I thought `GET _cluster/allocation/explain` only shows you information about an unassigned shard, but `"current_state": "started"` tells us that this shard is fine. You will need to ask it a more specific question. Find an unassigned shard with `GET _cat/shards` and then use this form of the allocation explain API: ```plaintext GET /_cluster/allocation/explain { "index": "myindex", "shard": 0, "primary": true } ``` ---
**Author:** ![drivera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drivera/32/39900_2.png) [@drivera](https://discuss.elastic.co/u/drivera)\ **Post date:** [February 28, 2019, 7:07pm UTC](https://discuss.elastic.co/t/health-check-on-my-es-cluster-shows-99-7/170082/6 "2019-02-28T19:07:17Z")
Looks like I have 7 unassigned shards. Running the explain API on the shard shows the output below. I notice, one of my data nodes is not in that list, I should have 5 but only see 4. Any ideas? { "index": "syslog-rsyslog-2019-02-25", "shard": 3, "primary": true, "current\_state": "started", "current\_node": { "id": "fCfm\_WQXTgeG52ca37Th1A", "name": "ist000247", "transport\_address": "10.44.0.188:9300", "weight\_ranking": 2 }, "can\_remain\_on\_current\_node": "yes", "can\_rebalance\_cluster": "no", "can\_rebalance\_cluster\_decisions": [ { "decider": "rebalance\_only\_when\_active", "decision": "NO", "explanation": "rebalancing is not allowed until all replicas in the cluster are active" }, { "decider": "cluster\_rebalance", "decision": "NO", "explanation": "the cluster has unassigned shards and cluster setting [cluster.routing.allocation.allow\_rebalance] is set to [indices\_all\_active]" } ], "can\_rebalance\_to\_other\_node": "no", "rebalance\_explanation": "rebalancing is not allowed, even though there is at least one node on which the shard can be allocated", "node\_allocation\_decisions": [ { "node\_id": "AAtmertMTeiuE\_i5CQTKAA", "node\_name": "ist000248", "transport\_address": "10.44.0.180:9300", "node\_decision": "yes", "weight\_ranking": 1 }, { "node\_id": "H8dbQ6dnQtSIOadYeqGz6w", "node\_name": "ist000245", "transport\_address": "10.44.0.157:9300", "node\_decision": "worse\_balance", "weight\_ranking": 2 }, { "node\_id": "YQXjt41cQrK7vDHuNXubZA", "node\_name": "ist000242", "transport\_address": "10.44.3.8:9300", "node\_decision": "worse\_balance", "weight\_ranking": 2 }, { "node\_id": "ihHNmtfqTGSugWowHxCILw", "node\_name": "ist000243", "transport\_address": "10.44.2.206:9300", "node\_decision": "worse\_balance", "weight\_ranking": 3 } ] } ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 1, 2019, 7:24am UTC](https://discuss.elastic.co/t/health-check-on-my-es-cluster-shows-99-7/170082/7 "2019-03-01T07:24:55Z")
It looks like this shard has a correctly allocated primary but is perhaps missing a replica? If so, you need to call the allocation explain API with `"primary": false` to find out about the shards other than the primary. ---
**Author:** ![drivera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drivera/32/39900_2.png) [@drivera](https://discuss.elastic.co/u/drivera)\ **Post date:** [March 1, 2019, 10:43pm UTC](https://discuss.elastic.co/t/health-check-on-my-es-cluster-shows-99-7/170082/8 "2019-03-01T22:43:52Z")
David, much obliged for your help. I was able to fix my cluster by doing the following steps: 1. Set replicas in bad indices to 0 2. PUT \_cluster/settings { "persistent": { "cluster.routing.allocation.enable": "none" } } 3. POST \_flush/synced 4. Stop ES on master and data nodes 5. Start ES on the master nodes first 6. Start ES on the data nodes second 7. Change “cluster.routing.allocation.enable”: “all” 8. Enable replica shards in bad indices to 1 I'm back to green! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 29, 2019, 10:43pm UTC](https://discuss.elastic.co/t/health-check-on-my-es-cluster-shows-99-7/170082/9 "2019-03-29T22:43:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.