ILM is failing to delete a few closed indices. We are getting the following issue:
policy [<policyName>] for index [<indexName>] on an error step due to a transient error, moving back to the failed step [wait-for-shard-history-leases] for execution. retry attempt [20077]
Cluster is health:
{
"cluster_name": "es",
"status": "green",
"timed_out": false,
"number_of_nodes": 9,
"number_of_data_nodes": 6,
"active_primary_shards": 1339,
"active_shards": 2678,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100.0
}
ILM policy:
"<policyName>": {
"version": 1,
"modified_date": "2023-06-28T17:44:22.693Z",
"policy": {
"phases": {
"delete": {
"min_age": "122d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
},
"in_use_by": {
"indices": [
<indicesToBeDeleted>
],
"data_streams": [],
"composable_templates": []
}
}
I couldn't find any explanation of what "wait-for-shard-history-leases" means in practice. According to other topics in this forum, ES is able to delete closed indices.
Can someone help?
Thank you.