I am having some issues with dangled indexes. Scenario seems to be:
*) Data node went down, was down for a while.
*) Life went on, some indexes were deleted from cluster.
*) Data node came back, saw indices in local filesystem, no longer in cluster, did the auto import of indices to cluster.
*) At this point I start to see my pending tasks fill slowly up (GET /_cluster/health?pretty). Lots of "allocation dangled indices" task.
Overall cluster performance degrades to the point that it becomes unusable.
Sometime ago there was an auto_import_dangled configuration to allow us to bypass this importing of dangled indexes. It seems to have disappeared in 6.3.1.
Any ideas on what I should to do either:
*) Avoid the dangled indices import in the first place?
*) Prevent the pending tasks to fill up with these events?
I know which data node is causing this issue, I could wipe out node data and restart that node and I'm guessing it would solve my issue. But it seems a bit extreme as all my data will need to be recreated. There has to be a way to have the data node rejoin without this issue no?