I have a situation where deleted indexes are not deleted from all nodes. As such, when ES is restarted, this Dangling Shard is imported as a Dangling Index and is thus UNASSIGNED and causes the cluster to enter "red" status and alarm bells start going off and things stop working.
I run filebeat/metricbeat and curator. Curator deletes all "metricbeat" indexes older than 30 days. It runs every night. My cluster is green for 'days' (not months..and this is a problem).
Then being already July, ES may restart for some reason (update of config or some other cycle) and then I get metricbeat indexes UNASSIGNED from january.
I then run curator again or manually delete these indexes from earlier, then the cluster goes green. (as mentioned in other tickets, curator wont necessarily see them if they are UNASSIGNED).
How can I run a program (which I am happy to do on all nodes) that would delete all index shards from the disk that don't have an index in ES? Or have an index, but are unassigned due to dangling_import? Any idea how to go about this? (need to do both, need to clean up the disk as ES is not deleting the files)
(By the way, saw the other topic regarding this from Feb 21...but, this question isn't about curator, this question is about how to I manually go about deleting these dangling indexes from the disk...they DO exist. If there is no fix for ES during index DELETE, then we need a workaround...so, how do we build the workaround?)