APM Indices growing endlessly despite ILM

[cawoodm (Marc)]

Our APM indices are growing without bound and extremely large despite a delete aftaer 30 days ILM policy.

Each index has a policy attached which say "rollover every 7 days" (which seems to be working) and delete "delete 30 days from rollover" which is not working. As you can see we have indices from last year laying around:

health status index                 id                     pri rep docs.count docs.deleted store.size creation.date.string
green  open   apm-7.3.1-span-000055 qUkLLtiuTFeIb4TxcpKsQA   1   0    5989569            0      1.4gb 2019-10-23T23:15:25.350Z
green  open   apm-7.3.1-span-000056 enjFAyiiQp6NDWQV8FSiCA   1   0    5984847            0      1.4gb 2019-10-24T23:25:25.240Z
green  open   apm-7.3.1-span-000057 wN_73OTZRECluFrk1nCu7w   1   0    5988669            0      1.4gb 2019-10-25T23:35:25.224Z
green  open   apm-7.3.1-span-000054 YIwExmcRQfmEQP62jcytsA   1   0    5987578            0      1.4gb 2019-10-22T23:05:25.326Z

WHY is ILM not deleting old indices?

When we run POST }/apm-7.3.1-span-000055/_ilm/retry we get "cannot retry an action for an index [apm-7.3.1-span-000055] that has not encountered an error when running a Lifecycle Policy".

[warkolm (Mark Walkom)]

Can you check what's happening with https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-explain-lifecycle.html?

[cawoodm (Marc)]

{
    "indices": {
        "apm-7.3.1-span-000055": {
            "index": "apm-7.3.1-span-000055",
            "managed": true,
            "policy": "apm-7.3.1-span",
            "lifecycle_date": "2019-10-24T23:25:25.499Z",
            "lifecycle_date_millis": 1571959525499,
            "age": "298.25d",
            "phase": "completed",
            "phase_time": "2019-10-31T23:35:25.481Z",
            "phase_time_millis": 1572564925481,
            "action": "completed",
            "action_time": "2019-10-31T23:35:25.481Z",
            "action_time_millis": 1572564925481,
            "step": "completed",
            "step_time": "2019-10-31T23:35:25.481Z",
            "step_time_millis": 1572564925481,
            "phase_execution": {
                "policy": "apm-7.3.1-span",
                "version": 1,
                "modified_date": "2019-08-30T14:53:42.563Z",
                "modified_date_in_millis": 1567176822563
            }
        }
    }
}

[warkolm (Mark Walkom)]

Ok, it might be worth telling it to retry?

[cawoodm (Marc)]

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "cannot retry an action for an index [apm-7.3.1-span-000055] that has not encountered an error when running a Lifecycle Policy"
            }
        ],
        "type": "illegal_argument_exception",
        "reason": "cannot retry an action for an index [apm-7.3.1-span-000055] that has not encountered an error when running a Lifecycle Policy"
    },
    "status": 400
}

[cawoodm (Marc)]

In Kibana when I click the index it says it has an ILM Policy assigned:

However when I view the list of Policies and search for this policy it shows 0 linked indices:

[cawoodm (Marc)]

It seems the policy IS associated with a template. Here is the template:

"apm-7.3.1-span": {
        "order": 2,
        "index_patterns": [
            "apm-7.3.1-span*"
        ],
        "settings": {
            "index": {
                "lifecycle": {
                    "name": "apm-7.3.1-span",
                    "rollover_alias": "apm-7.3.1-span"
                }
            }
        },
        "mappings": {},
        "aliases": {}
    }

[cawoodm (Marc)]

The index also seems to think it has an ILM policy

GET /apm-7.3.1-span-000055/_settings
{
  "apm-7.3.1-span-000055": {
      "settings": {
          "index": {
              "mapping": {
                  "total_fields": {
                      "limit": "2000"
                  }
              },
              "refresh_interval": "5s",
              "blocks": {
                  "write": "true"
              },
              "provided_name": "<apm-7.3.1-span-000055>",
              "creation_date": "1574830525330",
              "priority": "50",
              "number_of_replicas": "0",
              "version": {
                  "created": "7030099",
                  "upgraded": "7070099"
              },
              "lifecycle": {
                  "name": "apm-7.3.1-span",
                  "rollover_alias": "apm-7.3.1-span",
                  "indexing_complete": "true"
              },
              "codec": "best_compression",
              "number_of_shards": "1"
          }
      }
  }
}