# Checking elasticsearch health during rolling restart

**URL:** <https://discuss.elastic.co/t/checking-elasticsearch-health-during-rolling-restart/215023>\
**Category:** Elasticsearch\
**Created:** [January 14, 2020, 5:19pm UTC](https://discuss.elastic.co/t/checking-elasticsearch-health-during-rolling-restart/215023 "2020-01-14T17:19:33Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![lag13](https://avatars.discourse-cdn.com/v4/letter/l/e47774/32.png) [@lag13](https://discuss.elastic.co/u/lag13)\
**Post date:** [January 14, 2020, 5:19pm UTC](https://discuss.elastic.co/t/checking-elasticsearch-health-during-rolling-restart/215023/1 "2020-01-14T17:19:33Z")

</div>

Hello! When doing a rolling upgrade ([Rolling upgrades | Elasticsearch Guide [7.3] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/rolling-upgrades.html)), is checking that there are 0 initializing and relocating shards a reliable way to determine if you can restart the next node as opposed to checking that the "status" of the cluster is "green"?

**Background**

I'm writing a script to do a rolling upgrade on my 7.3.2 elasticsearch cluster deployed in AWS. For me that means (at a high level):

```
for each ec2 instance:
  1. terminate the instance
  2. wait for an ASG to spin up an instance to replace the terminated one
  3. wait for ES cluster to be "healthy" before terminating the next node

```

My question revolves around (3). The rolling upgrade documentation seems to suggest to first wait for the cluster to become "green" but if it doesn't then you can continue the rolling upgrade if there are no initializing or relocating shards. Instead of first checking that the cluster is "green" can I just check that there are no initializing or relocating shards? That would make the script logic simpler.

**Excerpt from the documentation** :

> Before upgrading the next node, wait for the cluster to finish shard allocation. You can check progress by submitting a \_cat/health request. Wait for the `status` column to switch from `yellow` to `green` . Once the node is `green` , all primary and replica shards have been allocated.  
> IMPORTANT:  
> During a rolling upgrade, primary shards assigned to a node running the new version cannot have their replicas assigned to a node with the old version. The new version might have a different data format that is not understood by the old version. If it is not possible to assign the replica shards to another node (there is only one upgraded node in the cluster), the replica shards remain unassigned and status stays `yellow` . In this case, you can proceed once there are no initializing or relocating shards (check the `init` and `relo` columns). As soon as another node is upgraded, the replicas can be assigned and the status will change to `green` .

Thanks for any and all advice!

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [January 14, 2020, 5:35pm UTC](https://discuss.elastic.co/t/checking-elasticsearch-health-during-rolling-restart/215023/2 "2020-01-14T17:35:08Z")

</div>

One issue that can happen with the first node. If Lucene is upgraded and a new index happens to get created, I think it allocates on the highest level Lucene nodes only. In my case, it refused to allocate it's replica on a lower-level Lucene node, so the cluster never goes green. Most of our indices are 1 shard, but I wonder if one tried to create more than one shard, I think it would fail.

This happened more frequently on my test cluster with quick rollover, so far, I've never seen it happen on production-like clusters.

Also, we are now upgrading all nodes on a rack at the same time, it's quicker than a node at a time :-).

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [January 14, 2020, 5:56pm UTC](https://discuss.elastic.co/t/checking-elasticsearch-health-during-rolling-restart/215023/3 "2020-01-14T17:56:46Z")

</div>

> [@rugenl](#):
>
> Also, we are now upgrading all nodes on a rack at the same time, it's quicker than a node at a time

I do same. shutdown everything and upgrade everything. done is less then 15 min top.  
But then my data are not mission critical.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 11, 2020, 5:56pm UTC](https://discuss.elastic.co/t/checking-elasticsearch-health-during-rolling-restart/215023/4 "2020-02-11T17:56:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.