Node subfolders have differnet indices

Elastic Stack Elasticsearch
1

We have a single node elasticsearch installation for ELK and today we noticed that all of our Kibana Dashboards are gone and only data from the last few days is being displayed.

Looking in the Elasticsearch data directories I found two sub-directories under nodes, elasticsearch/nodes/0 and elasticsearch/nodes/1. On our test environments they have the same indices, but on the production environment 0 has the dashboards and all of the data up to the 28th of Aug and 1 has only data since the 28th of Aug, which is what Kibana must be using.

Any idea how this might have happened and if there's anything I can do to fix it?

2

Multiple directories like that means you had multiple nodes on the same host, so some of that data may be in the 1 node.

Try starting up two nodes on that host and see if the data "returns".

3

Thanks, figured out the issue, looks like there was a zombie elasitcsearch running, making the live elasticsearch be counted as a second node. Killed stuff off I have dashboards back, now just the last 3 days of data is missing because its under the node/1 folder. Was thinking of copying the data to node/0, any tips on how to do that safely?

4

You can try copying the files directly to the other folder, it may or may not work though!

5

This worked for the most part. A few gaps in the data for days that existed in both folders.

Thanks!