# Restarting a Single Node in a Cluster Always Forces a Reallocation

**URL:** <https://discuss.elastic.co/t/restarting-a-single-node-in-a-cluster-always-forces-a-reallocation/149366>\
**Category:** Elasticsearch\
**Created:** [September 20, 2018, 10:26pm UTC](https://discuss.elastic.co/t/restarting-a-single-node-in-a-cluster-always-forces-a-reallocation/149366 "2018-09-20T22:26:11Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![zuhana2018](https://avatars.discourse-cdn.com/v4/letter/z/dbc845/32.png) [@zuhana2018](https://discuss.elastic.co/u/zuhana2018)\
**Post date:** [September 20, 2018, 10:26pm UTC](https://discuss.elastic.co/t/restarting-a-single-node-in-a-cluster-always-forces-a-reallocation/149366/1 "2018-09-20T22:26:11Z")

</div>

**Question**  
An ES 6.4 node restart does not persist the UUID and causes shard reallocations & sync's when the node rejoins the cluster. The is there another configuration other than path.data that need to be set to persist the UUID?

**Issue**  
When we restart a single ES Docker node, the indices are yellow until a reallocation takes place.  
However, I would expect that the restarted node to rejoin and prevent a reallocation. The node rejoins  
in a matter of seconds but the cluster remains unhealthy (yellow) until the node departed timeout.

> GET \_cluster/allocation/explain  
> ...  
> cannot allocate because the cluster is still waiting 9.1m for the departed node holding a replica to rejoin, despite being allowed to allocate the shard to at least one other node  
> ...

^ The node containing the shard(s) has already joined and is healthy (yellow).

The ES Docker container mounts a volume for the path.data in the same place on the host so that the data persists across restarts. I would expect the UUID to persist across reboots if the path.data is set correctly, but I see that after a reboot the UUID for the node changes:

> /\_cat/nodes?v&h=id,ip,node.role,master,name'  
> id ip node.role master name  
> jEmZ 172.27.1.73 mdi - 172.27.1.73  
> YL\_h 172.27.1.48 mdi - 172.27.1.48  
> -dST 172.27.1.124 mdi \* 172.27.1.124  
> cMXK 172.27.1.244 mdi - 172.27.1.244  
> j1qr 172.27.1.36 mdi - 172.27.1.36

I've tried to follow the guide for [ES Rolling Upgrades](https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html) but that doesn't seem to persist the UUID.

Is there something i'm missing?

Most configurations are set to default Elasticsearch 6.4. settings. This question is similar to what is described here [Delayed unassigned shards - #5 by chihccha](https://discuss.elastic.co/t/delayed-unassigned-shards/65400/5).

> GET \_cluster/health  
> {  
> "active\_primary\_shards": 30,  
> "active\_shards": 72,  
> "active\_shards\_percent\_as\_number": 80.0,  
> "cluster\_name": "test-cluster",  
> "delayed\_unassigned\_shards": 18,  
> "initializing\_shards": 0,  
> "number\_of\_data\_nodes": 5,  
> "number\_of\_in\_flight\_fetch": 0,  
> "number\_of\_nodes": 5,  
> "number\_of\_pending\_tasks": 0,  
> "relocating\_shards": 0,  
> "status": "yellow",  
> "task\_max\_waiting\_in\_queue\_millis": 0,  
> "timed\_out": false,  
> "unassigned\_shards": 18  
> }

---

<div class="post-metadata">

**Author:** ![zuhana2018](https://avatars.discourse-cdn.com/v4/letter/z/dbc845/32.png) [@zuhana2018](https://discuss.elastic.co/u/zuhana2018)\
**Post date:** [September 21, 2018, 10:38pm UTC](https://discuss.elastic.co/t/restarting-a-single-node-in-a-cluster-always-forces-a-reallocation/149366/2 "2018-09-21T22:38:22Z")

</div>

Turns out this was an issue with our configuration of Mesos (with Marathon framework) and not ElasticSearch.

We've been running the Mesos agent inside a Docker container. However, this does not work if we want to use Mesos's Persistent Volume feature since the mesos process in docker is unable to mount on the host filesystem.

This issue is resolved when after we moved Mesos to run on directly the host and be managed by systemd. The volumes are persisted and ES is able to restart without generating a new UUID.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 19, 2018, 10:38pm UTC](https://discuss.elastic.co/t/restarting-a-single-node-in-a-cluster-always-forces-a-reallocation/149366/3 "2018-10-19T22:38:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
