Hello all! I am pretty new to elasticsearch but was tasked with replacing our cluster’s data nodes that were about to hit disc capacity. I started with one and thought I had done due diligence in researching the safe way to do a replacement. I spun up a new VM to be the replacement, placed it in the cluster, and then set the data node to be replaced to be deallocated. After a while I noticed that deallocation had stopped before the old data node had actually reached zero shards. Digging around I discovered, to my horror, that I had overlooked version numbers when creating the new data node and that had led to cluster being unable to balance with shards that couldn’t be moved at all giving the error message target node version [XXX] is older than source node version [XXX]
Reading through documentation it seems that the official fix would be to do a rolling upgrade. That comes with all kinds of risks in terms of breaking changes, etc. I am pretty adverse to doing a rolling upgrade unless I absolutely have to. Do I have any other options here? Currently I have removed the old data node from the exclusion list and added the newer data node (the only one with the new ES version on it) to it so the newer node isn’t being assigned any more data but is sitting there with 50 shards on it that appear unable to be moved anywhere. Our cluster is green and I am not hearing any complaints from either devs nor clients at this time but need to fix this somehow. I’m wondering if I can’t simply go through the shards one at a time, delete them from the newer node, and count on ES to use replicas to replace those shards on one of the other machines.
Welcome to our community! 
Shards can move between versions equal to, or higher than whatever version they were created on. If you have replica shards you can try to remove the node with the higher version and add it back down a lower version of Elasticsearch. It should work, but it's not really a supported process as such.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.