Elasticsearch Docker container restart cause UUID to change

Elastic Stack Elasticsearch
1

Hi guys!
So 7.1.0 is out and we where really excited to use RBAC so I did a full cluster restart upgrade but now I'm stuck with an error on my data nodes saying that the local UUID do not match the cluster one. After a quick test I realize that by killing the Elastic-master pods to do my upgrade, the UUID change each time the pods are span. Is this the wanted behaviour ? Since the UUID is written on file for datanode, how can I make my data node follow the master UUID changes ?

Thanks

PS: Weird thing is when I was running 7.0.1, I did made pod restart time to time and naver face that issue ..

Dave

2

It sounds like you lost the data folders of all your master-eligible nodes and bootstrapped a fresh cluster. Note that the master nodes store metadata that is critical to the health of the cluster. Similar as to data nodes, you can't just wipe the master's data folder and expect the cluster to still be healthy. The following tool will help you get the data nodes connected again, but note that what was done here is operationally unsafe and might have resulted in data loss.

The reason why you might not have observed this previously is that you perhaps only restarted single pods (not all master pods at once), allowing ES to compensate for this.

3

Thanks Yannick I wasn't aware that master nodes relied on /data when there where dedicated entity in the stack. I've added a persistent volume for each of them and it's working great now :smiley:
Thank you very much

Dave

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.