Indices gone after upgrade (7.8.0 to 7.9.1)

[rcz (Martin Alstrup)]

Hello,

I'm in a bit of a pickle. A lot of my indices are gone after an upgrade.

I have a Logstash pumping in data into an alias.
The alias still pointed to the same index name after upgrading, but the old data was gone. It has just made a new index.
The indices were rolled every 2 days, and all the previous indices are gone as well.
Not all indices are gone, but I can't find any managed indices from the past few months.

I'm fairly confident that the raw files are still there, since I have about 2.5 TB of index data distributed across my 3 nodes.
I don't think anything was deleted.

Is there any way to recover indices from disk?
Any help is greatly apprecieated!

Thanks,
Martin

[warkolm (Mark Walkom)]

How did you do the upgrade?

[rcz (Martin Alstrup)]

Hi Mark. I did the upgrade as I've done many times (per node):

1. Disable shard reallocation
2. Upgrade RPM package
3. Restart Elasticsearch service
4. Enable shard reallocation
5. Wait for cluster to report green health

It is a 3 node cluster with each node having all features enabled (all masters).
Is it possible to recover indices from disk?

[warkolm (Mark Walkom)]

Can you try using https://www.elastic.co/guide/en/elasticsearch/reference/7.9/dangling-indices-list.html?

[rcz (Martin Alstrup)]

There seems to be no dangling indices:

# GET /_dangling
{
  "_nodes" : {
    "total" : 3,
    "successful" : 3,
    "failed" : 0
  },
  "cluster_name" : "clm",
  "dangling_indices" : [ ]
}

[rcz (Martin Alstrup)]

Hi again @warkolm,

I found backups of the data directories of each node. They are full filesystem backups. Now I'm wondering which method would be best for restoring those backups.

First I thought I could simply copy all the files (nodes directory) over, but now I'm worried what will happen as the index names will probably clash - at least for the newest index of the lost data/first index of new data..?

Another option I see would be to start up a single-node ES cluster with the combined data directories of the 3 backups and then transfer the old indices through snapshots.
Or maybe it's better to make snapshots of the new data, fully restore the backups, and then restore the snapshots.

What would you suggest?
Any help is greatly appreciated!