Our APM indices are growing without bound and extremely large despite a delete aftaer 30 days ILM policy.
Each index has a policy attached which say "rollover every 7 days" (which seems to be working) and delete "delete 30 days from rollover" which is not working. As you can see we have indices from last year laying around:
health status index id pri rep docs.count docs.deleted store.size creation.date.string green open apm-7.3.1-span-000055 qUkLLtiuTFeIb4TxcpKsQA 1 0 5989569 0 1.4gb 2019-10-23T23:15:25.350Z green open apm-7.3.1-span-000056 enjFAyiiQp6NDWQV8FSiCA 1 0 5984847 0 1.4gb 2019-10-24T23:25:25.240Z green open apm-7.3.1-span-000057 wN_73OTZRECluFrk1nCu7w 1 0 5988669 0 1.4gb 2019-10-25T23:35:25.224Z green open apm-7.3.1-span-000054 YIwExmcRQfmEQP62jcytsA 1 0 5987578 0 1.4gb 2019-10-22T23:05:25.326Z
WHY is ILM not deleting old indices?
When we run
POST }/apm-7.3.1-span-000055/_ilm/retry we get
"cannot retry an action for an index [apm-7.3.1-span-000055] that has not encountered an error when running a Lifecycle Policy".