Upgrade to 8.1.2 failing with unexpected folder encountered during data folder upgrade

Elastic Stack Elasticsearch
3

Thank you very much @DavidTurner. It is odd since the host is exclusively an ES node. Further the data of ES is written to another SSD from the OS, meaning the /mnt/ should have no other data written even from the OS.

I do reckon this has something to do with last years disk failure that you helpmed me diagnose in this thread (the month of the thread matches :expressionless: - Node sync fails and cluster goes to "red" - #21 by parthmaniar).

I will need to replace the SSD meaning the new one will have no data (but the Elasticsearch settings and the OS will remain as is). Hence, is following feasable for recovery?:

  1. Attach a new SSD to the VM with the same mounth path as the previous one.
  2. Upgrade last remaining ES data node to 8.1.2. This node has all of the data.
  3. I have turned off routing of data:
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}
  1. I can eable routing of data which would fill up the new disk? Will this work?

The reason I am stuck is:

  1. One data & master node is on 7.17.2 and it has all of the data intact.
  2. One data & master node is on 8.1.2 but has the data disk (data folder for ES) that has failed
  3. Third is a voting only node that has upgraded successfully.

Thank you very much.