Delete old indexes via Index lifecycle policy doesn't work because of shards error

[mfa]

I have read through the docs and some posts here, however my delete policy is not working as expected. Here is my policy:

    PUT _ilm/policy/deleting_policy
    {
      "policy": {
        "phases": {
          "hot": {
            "min_age": "0ms",
            "actions": {
              "rollover": {
                "max_size": "50gb"
              },
              "set_priority": {
                "priority": 100
              }
            }
          },
          "delete": {
            "min_age": "30d",
            "actions": {
              "delete": {
                "delete_searchable_snapshot": true
              }
            }
          }
        }
      }
    }

And the index setting is as following:

    {
      "index.blocks.read_only_allow_delete": "false",
      "index.query.default_field": [
        "*"
      ],
      "index.refresh_interval": "1s",
      "index.write.wait_for_active_shards": "1",
      "index.lifecycle.name": "deleting_policy",
      "index.lifecycle.rollover_alias": "myindex-alias",
      "index.priority": "100",
      "index.number_of_replicas": "1"
    }

Here is some parts of the index settings regarding shards (which was the default setting):

{
  "settings": {
    "index": {
      "lifecycle": {
        "name": "deleting_policy",
        "rollover_alias": "trafficinfo-alias"
      },
      "number_of_shards": "1",
      "provided_name": "trafficinfo-000001",
      "creation_date": "1605959828528",
      "priority": "100",
      "number_of_replicas": "1",
      "uuid": "qc0PGsh-TTmcXSN-LMW6XQ",
      "version": {
        "created": "7090399"
      }
    }
  },
####
"number_of_routing_shards": "1",
      "write": {
        "wait_for_active_shards": "1"
      },
####
"allocation": {
          "enable": "all",
          "total_shards_per_node": "-1"
        }
####
"allocation": {
        "max_retries": "5",
        "existing_shards_allocator": "gateway_allocator"
      }

The delete-policy started to apply on an index (myindex1-000001) with size:357gb and it has created a rollover index: myindex1-000002 with status "index is red; not all primary shards are active"
and myindex1-000001 lifecycle action status became "waiting for [DEFAULT] shards to become active, but only [0] are active"

I also see under elastic-cloud deployment this error:
"Unhealthy deployment
7 Elasticsearch shards unavailable"
what is it missing here?

The java application is pointing to index name: myindex1-000001 how it's gonna know about the rollover index with name "myindex1-000002"?

I'm wondering what I am missing here that the deleting policy doesn't work well?

My instance went out of the storage so I need to remove some data as soon as possible in order to get back my instance to healthy status.
Any help would be appreciated.

[dakrone (Lee Hinman)]

Could you share the output of GET /<your-index>/_ilm/explain?human here?