ILM is failing to delete closed indices

Elastic Stack Elasticsearch
1

ILM is failing to delete a few closed indices. We are getting the following issue:

policy [<policyName>] for index [<indexName>] on an error step due to a transient error, moving back to the failed step [wait-for-shard-history-leases] for execution. retry attempt [20077]

Cluster is health:

{
    "cluster_name": "es",
    "status": "green",
    "timed_out": false,
    "number_of_nodes": 9,
    "number_of_data_nodes": 6,
    "active_primary_shards": 1339,
    "active_shards": 2678,
    "relocating_shards": 0,
    "initializing_shards": 0,
    "unassigned_shards": 0,
    "delayed_unassigned_shards": 0,
    "number_of_pending_tasks": 0,
    "number_of_in_flight_fetch": 0,
    "task_max_waiting_in_queue_millis": 0,
    "active_shards_percent_as_number": 100.0
}

ILM policy:

"<policyName>": {
        "version": 1,
        "modified_date": "2023-06-28T17:44:22.693Z",
        "policy": {
            "phases": {
                "delete": {
                    "min_age": "122d",
                    "actions": {
                        "delete": {
                            "delete_searchable_snapshot": true
                        }
                    }
                }
            }
        },
        "in_use_by": {
            "indices": [
                <indicesToBeDeleted>
            ],
            "data_streams": [],
            "composable_templates": []
        }
    }

I couldn't find any explanation of what "wait-for-shard-history-leases" means in practice. According to other topics in this forum, ES is able to delete closed indices.

Can someone help?

Thank you.

1 Like
2

Why the indices are closed?

3

They are closed because of some internal policies of our application.
Does it matter? According to our understanding of ILM, they should be deleted by the ILM policy anyway.

4

Yes it matters. Closed indices remove can cause issue. Here are some similar discussions:

  1. Elasticsearch ILM failed delete closed indices
  2. Index Lifecycle Management - closing indices

If you send close index request via application, It'd be better to remove or open the index via application again.

But the main question is "why you need to close indices?" like @Aaron mentioned in here.

5

OKay

6

These indices shouldn't be closed. This is a result of a not gracefully aborted operation. In a normal situation, we don't have closed indices.

However, given the possibility of the exceptional scenario above, we were expecting ILM to be able to delete and therefore clean up these closed indices, as referenced in Elasticsearch ILM failed delete closed indices - #2 by rtwolfe94022 : "Yes, ILM can delete closed indices"

7

That answer was provided by an non-Elastic volunteer so I am not sure why you are assuming that is correct.

Which version of Elasticsearch are you using?

8

I opened this topic to double-check that assumption.

Which version of Elasticsearch are you using?
8.9.1

So, can I assume ILM cannot delete closed indices?

9

I do not know. It is probably best to wait for someone from Elastic to weigh in on the topic as it is not covered in the docs.