we managed to delete our cluster. And we have no good backup.(I know that we fucked up )
Well, to be more precise, ES decided to delete all data after we started 3 new master nodes and assigned the data nodes to the new cluster(as it turned out this was a very bad decision)
1.5TB of data was gone within seconds/minutes. All acts of restoring the data failed.
I don't really care about the logs that we lost. We log enough and only really care about live data anyways.
But we miss our dashboards. Or at least some of them, since some guys made backups themselves.
We have a daily backup of the elasticsearch servers disk. I managed to restore this backup partially.
The index was previously stored in 5 shards with one replica, across 6 nodes. One node backup failed, so i only have the data from 5 nodes.
I managed to find all 5 shards in the backups (and some replicas), but the backup was not run on all servers at the exact same time.
For example: node1 backup was run at 8am, node2 backup maybe at 10am and so on.
The index was not changed between the backup times.
Putting the shards in a new ES cluster does not work.
Elasticsearch does not even attempt to load them.
Is there any way to somehow get the data? I just want our kibana dashboards back, because we put a lot of work in them.
a lecture regarding elasticsearch backups is not necessary. I know that we made a big mistake and we payed for it.
If there is any way this might work, i would be very happy.