I have read through the docs and some posts here, however my delete policy is not working as expected. Here is my policy:
PUT _ilm/policy/deleting_policy
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_size": "50gb"
},
"set_priority": {
"priority": 100
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}
And the index setting is as following:
{
"index.blocks.read_only_allow_delete": "false",
"index.query.default_field": [
"*"
],
"index.refresh_interval": "1s",
"index.write.wait_for_active_shards": "1",
"index.lifecycle.name": "deleting_policy",
"index.lifecycle.rollover_alias": "myindex-alias",
"index.priority": "100",
"index.number_of_replicas": "1"
}
Here is some parts of the index settings regarding shards (which was the default setting):
{
"settings": {
"index": {
"lifecycle": {
"name": "deleting_policy",
"rollover_alias": "trafficinfo-alias"
},
"number_of_shards": "1",
"provided_name": "trafficinfo-000001",
"creation_date": "1605959828528",
"priority": "100",
"number_of_replicas": "1",
"uuid": "qc0PGsh-TTmcXSN-LMW6XQ",
"version": {
"created": "7090399"
}
}
},
####
"number_of_routing_shards": "1",
"write": {
"wait_for_active_shards": "1"
},
####
"allocation": {
"enable": "all",
"total_shards_per_node": "-1"
}
####
"allocation": {
"max_retries": "5",
"existing_shards_allocator": "gateway_allocator"
}
The delete-policy started to apply on an index (myindex1-000001) with size:357gb and it has created a rollover index: myindex1-000002 with status "index is red; not all primary shards are active"
and myindex1-000001 lifecycle action status became "waiting for [DEFAULT] shards to become active, but only [0] are active"
I also see under elastic-cloud deployment this error:
"Unhealthy deployment
7 Elasticsearch shards unavailable"
what is it missing here?
The java application is pointing to index name: myindex1-000001 how it's gonna know about the rollover index with name "myindex1-000002"?
I'm wondering what I am missing here that the deleting policy doesn't work well?
My instance went out of the storage so I need to remove some data as soon as possible in order to get back my instance to healthy status.
Any help would be appreciated.