[ES5.6] Unassigned Shards due to DANGLING_INDEX_IMPORTED

Elastic Stack Elasticsearch
1

We are seeing multiple cases where an index was deleted > 1 month ago but after a node restart the shards of this index becomes unassigned due to DANGLING_INDEX_IMPORTED.

Actually, the corresponding index uuuid folder isn't getting deleted from a node's data directory.

I have the following questions:

  1. How can I ensure that the index folder is deleted from each of the cluster nodes after an index has been deleted.
  2. Is there an ES recommended way for deleting indices?
  3. Can this happen because an index was not closed before deleting it?
  4. How can I prevent this from happening again?
2

One thing I believe can cause this is if your cluster is not correctly configured and can suffer from split-brain scenarios. Please make sure you have followed these guidelines as this otherwise can/will lead to data loss.

This should be automatic in a correctly configured cluster.

Always use the APIs.

No.

3

In our current setup, all nodes are dataNodes as well as master eligible.
We've set min_master_nodes to n/2+1 and have our own custom algo to ensure split-brain never occurs.
Is there a way to disable import of dangling indexes?

4

Can you provide some additional details on this?

5

I'm not sure if I can disclose the details. Apologies!

6

I am not entirely sure there are no other causes, but the fact that you end up with dangling indicxes in a running cluster could indicate that your custom algorithm is not working properly under all conditions and failure scenarios...

Why are you not using the default algorithm?

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.