Currently working out some issues on our ELK deployment related to data loss and corrupted snapshots. I've since fixed the snapshot but previous ones are essentially lost.
Now, for context, we've had to remove the data_content
role from some machines. We did not go through the documented steps and ran elasticsearch-node repurpose
to delete the unwanted indices. We had thought at the time that restoring snapshots was the way we'd restore them, but turns out they were corrupted. Now we're trying in the short term to restore all functionality and learn from our mistakes.
Current issue is, stack monitoring isn't working in kibana due to a large number of task not found
errors.
I don't know where to go from here. Task indices are probably missing, but restarting the elastic cluster and kibana doesn't lead to a working state. Is there anything I should be cleaning up, or configuring otherwise?