Over the weekend, the 3 master nodes in a 7 node cluster would not start up due to lack of space in the data path and the logs path. Only one of the mast nodes came up. But the data nodes started to complain about bad state files.
I had to relocate the data and log paths (due to system issues(vormetric/encryption/security) I could not copy the data folder to new location). Once relocated, the master nodes came up but, the data nodes would not. They started to complain about state files.
A couple of questions :
- Can state be recovered in such a situation ?
- If I deleted the corrupt state files, would they be recreated ?
- if they cannot be recreated, can the data be recovered ?
Any help/ insight into this is much appreciated.
ES version 5.2
running on 3 Master Nodes (VMs), 4 data Nodes and 4 client nodes running on 4 servers.
Kibana 5.2 running on the data node servers as well.