Dangling indices :- coordinating node

[praveen.vemuri]

Hi, My cluster turned red due to dangling indexes.

i saw below warning message on the coordinating nodes. none of the nodes were restarted during this time. all the nodes in cluster were up and running from 20 days and this issue happened today.

can not be imported as a dangling index, as an index with the same name and UUID exist in the index tombstones. This situation is likely caused by copying over the data directory for an index that was previously deleted.

[praveen.vemuri]

I am seeing this issue immediately after curator ran to delete old indices.

[theuntergeek (Aaron Mildenstein)]

Curator is really just an index selection wrapper around the associated Elasticsearch APIs. A delete_indices in Curator is still just a DELETE in Elasticsearch. This means that you could have executed the same delete in Console in Kibana and ended up with dangling indices.

I suggest you look in the Elasticsearch log files to see if there are problems indicated there, with clustering, master nodes changing, etc.

[praveen.vemuri]

Thanks for your reply. i read your comment on other similarpost :).
i don't mean to say curator is causing the issue. i just observed that this is happening after the deletion of the old indices and want to update here.

may be cluster is trying to update the difference in cluster state as curator deleted few indices and one of the node has different cluster state?

[theuntergeek (Aaron Mildenstein)]

I'm not being defensive. It's just that the cluster state should never be out of sync between nodes, whether the DELETE is triggered by Curator or some other API call.

The cluster state is kept in sync by the elected master node. If you have a node that's out of sync, that's bad. Every delete you submit, regardless of which node it goes to, will trigger a cluster state update across all nodes. If one is not getting this update in a timely manner (milliseconds), that's a problem. This is why I suggested looking into the Elasticsearch logs.