# APM Indices growing endlessly despite ILM

**URL:** <https://discuss.elastic.co/t/apm-indices-growing-endlessly-despite-ilm/243846>\
**Category:** Elasticsearch\
**Tags:** ilm-index-lifecycle-management\
**Created:** [August 5, 2020, 10:09am UTC](https://discuss.elastic.co/t/apm-indices-growing-endlessly-despite-ilm/243846 "2020-08-05T10:09:11Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [August 5, 2020, 10:09am UTC](https://discuss.elastic.co/t/apm-indices-growing-endlessly-despite-ilm/243846/1 "2020-08-05T10:09:12Z")

</div>

Our APM indices are growing without bound and extremely large despite a delete aftaer 30 days ILM policy.

Each index has a policy attached which say "rollover every 7 days" (which seems to be working) and delete "delete 30 days from rollover" which is not working. As you can see we have indices from last year laying around:

```
health status index id pri rep docs.count docs.deleted store.size creation.date.string
green open apm-7.3.1-span-000055 qUkLLtiuTFeIb4TxcpKsQA 1 0 5989569 0 1.4gb 2019-10-23T23:15:25.350Z
green open apm-7.3.1-span-000056 enjFAyiiQp6NDWQV8FSiCA 1 0 5984847 0 1.4gb 2019-10-24T23:25:25.240Z
green open apm-7.3.1-span-000057 wN_73OTZRECluFrk1nCu7w 1 0 5988669 0 1.4gb 2019-10-25T23:35:25.224Z
green open apm-7.3.1-span-000054 YIwExmcRQfmEQP62jcytsA 1 0 5987578 0 1.4gb 2019-10-22T23:05:25.326Z

```

WHY is ILM not deleting old indices?

When we run `POST }/apm-7.3.1-span-000055/_ilm/retry` we get `"cannot retry an action for an index [apm-7.3.1-span-000055] that has not encountered an error when running a Lifecycle Policy"`.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 9, 2020, 11:18pm UTC](https://discuss.elastic.co/t/apm-indices-growing-endlessly-despite-ilm/243846/2 "2020-08-09T23:18:06Z")

</div>

Can you check what's happening with [https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-explain-lifecycle.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-explain-lifecycle.html)?

---

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [August 18, 2020, 5:27am UTC](https://discuss.elastic.co/t/apm-indices-growing-endlessly-despite-ilm/243846/3 "2020-08-18T05:27:00Z")

</div>

```
{
    "indices": {
        "apm-7.3.1-span-000055": {
            "index": "apm-7.3.1-span-000055",
            "managed": true,
            "policy": "apm-7.3.1-span",
            "lifecycle_date": "2019-10-24T23:25:25.499Z",
            "lifecycle_date_millis": 1571959525499,
            "age": "298.25d",
            "phase": "completed",
            "phase_time": "2019-10-31T23:35:25.481Z",
            "phase_time_millis": 1572564925481,
            "action": "completed",
            "action_time": "2019-10-31T23:35:25.481Z",
            "action_time_millis": 1572564925481,
            "step": "completed",
            "step_time": "2019-10-31T23:35:25.481Z",
            "step_time_millis": 1572564925481,
            "phase_execution": {
                "policy": "apm-7.3.1-span",
                "version": 1,
                "modified_date": "2019-08-30T14:53:42.563Z",
                "modified_date_in_millis": 1567176822563
            }
        }
    }
}
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 18, 2020, 5:28am UTC](https://discuss.elastic.co/t/apm-indices-growing-endlessly-despite-ilm/243846/4 "2020-08-18T05:28:42Z")

</div>

Ok, it might be worth telling it to retry?

---

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [August 18, 2020, 8:47am UTC](https://discuss.elastic.co/t/apm-indices-growing-endlessly-despite-ilm/243846/5 "2020-08-18T08:47:25Z")

</div>

```
{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "cannot retry an action for an index [apm-7.3.1-span-000055] that has not encountered an error when running a Lifecycle Policy"
            }
        ],
        "type": "illegal_argument_exception",
        "reason": "cannot retry an action for an index [apm-7.3.1-span-000055] that has not encountered an error when running a Lifecycle Policy"
    },
    "status": 400
}
```

---

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [August 18, 2020, 8:50am UTC](https://discuss.elastic.co/t/apm-indices-growing-endlessly-despite-ilm/243846/6 "2020-08-18T08:50:59Z")

</div>

In Kibana when I click the index it says it has an ILM Policy assigned:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/4/844375c0eb98a8ddf6f75a6a70dad76bd6042b8a.png)

However when I view the list of Policies and search for this policy it shows 0 linked indices:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/9/29a509100aaef73fbfcb986cafec08875466dc01.png)

---

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [August 18, 2020, 8:52am UTC](https://discuss.elastic.co/t/apm-indices-growing-endlessly-despite-ilm/243846/7 "2020-08-18T08:52:37Z")

</div>

It seems the policy IS associated with a template. Here is the template:

```
"apm-7.3.1-span": {
        "order": 2,
        "index_patterns": [
            "apm-7.3.1-span*"
        ],
        "settings": {
            "index": {
                "lifecycle": {
                    "name": "apm-7.3.1-span",
                    "rollover_alias": "apm-7.3.1-span"
                }
            }
        },
        "mappings": {},
        "aliases": {}
    }
```

---

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [August 18, 2020, 8:55am UTC](https://discuss.elastic.co/t/apm-indices-growing-endlessly-despite-ilm/243846/8 "2020-08-18T08:55:27Z")

</div>

The index also seems to think it has an ILM policy

```
GET /apm-7.3.1-span-000055/_settings
{
  "apm-7.3.1-span-000055": {
      "settings": {
          "index": {
              "mapping": {
                  "total_fields": {
                      "limit": "2000"
                  }
              },
              "refresh_interval": "5s",
              "blocks": {
                  "write": "true"
              },
              "provided_name": "<apm-7.3.1-span-000055>",
              "creation_date": "1574830525330",
              "priority": "50",
              "number_of_replicas": "0",
              "version": {
                  "created": "7030099",
                  "upgraded": "7070099"
              },
              "lifecycle": {
                  "name": "apm-7.3.1-span",
                  "rollover_alias": "apm-7.3.1-span",
                  "indexing_complete": "true"
              },
              "codec": "best_compression",
              "number_of_shards": "1"
          }
      }
  }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 15, 2020, 8:55am UTC](https://discuss.elastic.co/t/apm-indices-growing-endlessly-despite-ilm/243846/9 "2020-09-15T08:55:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.