Good day,
I have an Elasticsearch 6.8.1 single node cluster running inside a kubernetes statefulset to index documents that I need to look after and I noticed that the snapshot path.repo had not been setup previously.
So I added a folder for snapshots with the correct rights on the NFS server where the elastic data currently resides, created a persistent volume bound to the NFS directory I created and changed the statefulset to claim against the persistent volume, and mount it to the container.
I created the repository and started to snapshot the cluster inside the pod - happy days. I've done this for a few of the instances running inside the pods now without a problem.
However, I just ran into into the first situation when I brought the statefulset container back up after adding the path.repo to the config and wanted to start the snapshots, where I have 25 indices normally, I now only have 2 i.e. some indices missing.
Looking on Kibana I can see they are not there, however looking on the NFS data share itself which is bound to a different persistent volume for the data, all the data is still on disk and in the PV.
What can I do to get the indices back?
Prior to setting up the snapshots I stopped all nfs services, pods, and drained the kubernetes node to try and facilitate a state where nothing can touch the location of the cluster data seeing that it is always in a dynamic state.
I then created an atomic disk image using Acronis.
So the question is can I get those back, how to go about it etc. Any help greatly appreciated
Kind Regards
