Bad version on master node

Elastic Stack Elasticsearch
1

Hi everyone,

I have an ELASTIC cluster with 3 master nodes and 6 data nodes.

I'm having a problem because one master node has upgraded to 8.17.3 while all the other nodes are running 8.17.0.

I want to remove this master node and then reinstall it with the correct version.

The advantage is that it has no shards and is not active. Could you please guide me on how to proceed?

And possibly upgrade it to a data node, as 3 master nodes aren't necessarily necessary.

Thank you very much in advance.

2

This is incorrect. You need at least 3 master eligible nodes.

Have never downgraded a master node so will leave it for others. Why not upgrade all the other nodes to 8.17.3 instead?

3

Split-brain, it's correct, sorry.

Because I thought it would be faster to stabilize the cluster.
Because updates require stopping the nodes? Or is just an apt upgrade enough?

4

You need to restart the nodes.

You should upgrade the remaining master nodes and then all the other nodes, all Elasticsearch nodes needs to be on the same version.

Is the master node on version 8.17.3 the active master? I'm not sure if it is still possible to remove this node and re-add it on version 8.17.0, downgrades are not supported in Elasticsearch.

5

Nop not the active master. So I update all package of the nodes (apt update/upgrade && restart elasticsearch), and I begin by data node?
I need to do something with the shards or just update/restart? Thanks to you.

6

The question is, should I temporarily exclude the node I'm going to update from shard allocations or is this not mandatory? And do I do all the data to end up with the master?

7

So, in the first time, I need to stop the only master node in 8.17.3 to prevent problem?

8

I'm not sure, as mentioned I don't know if you can just remove this node on version 8.17.3 and recreate it in version 8.17.0.

This is not something that is documented as downgrades are not supported, the safest option would be to upgrade the cluster to the same version, for this you can follow the rolling upgrade documentation here.

1 Like
9

The concern would be he can’t follow the ordering suggested by the documentation. So he’s in uncharted waters. Undocumented anyways.

One option would just to shutdown the 8.17.3 master. Temporarily. Upgrade everything else, including the other nasters. Maybe add another master temporarily too. But it all carries sone risk.