Deleted cluster,cant restore index from non-ES snapshot (disk backup)

Elastic Stack Elasticsearch
1

Hello,

we managed to delete our cluster. And we have no good backup.(I know that we fucked up :slight_smile: )
Well, to be more precise, ES decided to delete all data after we started 3 new master nodes and assigned the data nodes to the new cluster(as it turned out this was a very bad decision)
1.5TB of data was gone within seconds/minutes. All acts of restoring the data failed.
I don't really care about the logs that we lost. We log enough and only really care about live data anyways.

But we miss our dashboards. Or at least some of them, since some guys made backups themselves.

We have a daily backup of the elasticsearch servers disk. I managed to restore this backup partially.

The index was previously stored in 5 shards with one replica, across 6 nodes. One node backup failed, so i only have the data from 5 nodes.

I managed to find all 5 shards in the backups (and some replicas), but the backup was not run on all servers at the exact same time.
For example: node1 backup was run at 8am, node2 backup maybe at 10am and so on.
The index was not changed between the backup times.

Putting the shards in a new ES cluster does not work.
Elasticsearch does not even attempt to load them.

Is there any way to somehow get the data? I just want our kibana dashboards back, because we put a lot of work in them.

a lecture regarding elasticsearch backups is not necessary. I know that we made a big mistake and we payed for it.

If there is any way this might work, i would be very happy.

Luca

2

The backups should run at the same time across all nodes, so it's odd that you have different times there.

Are you sure the data is being placed back into where ever path.data points to?

3

I don't think that you understood me correctly.
We did not use any kind of elasticsearch backup. A full disk backup is made once every day. But this is not done at the same time on all nodes.

Yes, the directory is correct. After altering the _state files, elasticsearch tries to load them but tells me that some checksums are wrong and that the shards are corrupted.

4

Ohh ok. Then yeah its probably a mismatch somewhere.
I'm not sure what you can do here, but perhaps a lucene level expert can give you something to extract the data. No promises though!

5

Thank you for trying to help

6

Take a look at Luke https://code.google.com/p/luke/. It's a tool for diag, repair Lucene indexes.

7

Thanks, I will give it a try and report back (might take a few days, for different reasons)

8

okay, it looks promising so far.
I managed to open each shards index directory in the tool you mentioned.

Do you have any experience using luke?

What would be the next step to get this stuff back into elasticsearch?

9

If found a Solution!!

The master also needed a /_state/.st
This is why ES did not load anything. It did not know that an index was there.
Fortunately i also had a backup of the _state file and nothing changed there inbetween backups.

I am so happy right now.. :smiley:

Thanks so much for helping. Even though the tool itself was not the solution, it helped me undestand everything better and finally led to a solution (after countless google searches)

1 Like