Delete old indexes via Index lifecycle policy doesn't work because of shards error

I have read through the docs and some posts here, however my delete policy is not working as expected. Here is my policy:

    PUT _ilm/policy/deleting_policy
    {
      "policy": {
        "phases": {
          "hot": {
            "min_age": "0ms",
            "actions": {
              "rollover": {
                "max_size": "50gb"
              },
              "set_priority": {
                "priority": 100
              }
            }
          },
          "delete": {
            "min_age": "30d",
            "actions": {
              "delete": {
                "delete_searchable_snapshot": true
              }
            }
          }
        }
      }
    }

And the index setting is as following:

    {
      "index.blocks.read_only_allow_delete": "false",
      "index.query.default_field": [
        "*"
      ],
      "index.refresh_interval": "1s",
      "index.write.wait_for_active_shards": "1",
      "index.lifecycle.name": "deleting_policy",
      "index.lifecycle.rollover_alias": "myindex-alias",
      "index.priority": "100",
      "index.number_of_replicas": "1"
    }

Here is some parts of the index settings regarding shards (which was the default setting):

{
  "settings": {
    "index": {
      "lifecycle": {
        "name": "deleting_policy",
        "rollover_alias": "trafficinfo-alias"
      },
      "number_of_shards": "1",
      "provided_name": "trafficinfo-000001",
      "creation_date": "1605959828528",
      "priority": "100",
      "number_of_replicas": "1",
      "uuid": "qc0PGsh-TTmcXSN-LMW6XQ",
      "version": {
        "created": "7090399"
      }
    }
  },
####
"number_of_routing_shards": "1",
      "write": {
        "wait_for_active_shards": "1"
      },
####
"allocation": {
          "enable": "all",
          "total_shards_per_node": "-1"
        }
####
"allocation": {
        "max_retries": "5",
        "existing_shards_allocator": "gateway_allocator"
      }

The delete-policy started to apply on an index (myindex1-000001) with size:357gb and it has created a rollover index: myindex1-000002 with status "index is red; not all primary shards are active"
and myindex1-000001 lifecycle action status became "waiting for [DEFAULT] shards to become active, but only [0] are active"

I also see under elastic-cloud deployment this error:
"Unhealthy deployment
7 Elasticsearch shards unavailable"
what is it missing here?

The java application is pointing to index name: myindex1-000001 how it's gonna know about the rollover index with name "myindex1-000002"?

I'm wondering what I am missing here that the deleting policy doesn't work well?

My instance went out of the storage so I need to remove some data as soon as possible in order to get back my instance to healthy status.
Any help would be appreciated.

Could you share the output of GET /<your-index>/_ilm/explain?human here?

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.