I have indices per day (e.g. myindex_yyyyMMdd) and under normal operating conditions it appears my rolling window(s) are working just fine and with all nodes connected to my cluster, deletes execute successfully and the cluster status and master logs reflect the deletion.
However I've seen under two different scenarios (both involving taking a previously active/healthy node offline and bringing it back online) where long since deleted indices (well outside my rolling window) come back as dangling indices. In this interesting scenario where the index has been deleted, the cluster is unable to fully recover (no surprise there, the shards should actually be deleted) and of course my cluster stays in the red until the dangling indices are deleted.
I have a handful of clusters all running ElasticSearch 1.7.1, all of which show this behavior from time to time but not consistently.
I've seen this happen both with full cluster restarts (shut down the entire cluster) as well as individual node restarts.
I fully understand the dandling index concept, particularly around nodes that may have been offline when the deletions had occurred but not for nodes that were active/healthy at the time of deletion.
Any thought/input would be appreciated.