I've had an elasticsearch cluster running since late June. My search wasn't working today and when I looked at the app I was getting this error:
elasticsearch.exceptions.NotFoundError: TransportError(404, 'index_not_found_exception', 'no such index')
When I went to Kibana, other than saying my license had expired (for x-pack, monitoring, etc) ... there was no messages. It was asking me to create a new index because it couldn't find any ... the only one it found was for an index called README.
When I ran LSBLK, I could see the drives were still mounted but only had 194 MB of data (used to be close to 300-500gb). All 3 nodes don't have their data anymore. It's just gone.
When I looked at the server's keyboard commands nothing unusual and the login activity shows it was only me. Does anyone have any idea how I can find out where the data went or how I can recover it?
I looked at the cluster health and it shows that there are 3 unassigned shards ... but with all 3 nodes online and with empty hard drives where could they be? I have always had automatic shard allocation enabled.
... There has been NO data recently indexed and I'm sure no one our end deleted the index or altered it any way ... we definitely didn't accidentally delete the index from the command line. There have only been queries to the index using the python elasticsearch client.