# These old indices should be deleted by ES according to ILM policy, right?

**URL:** <https://discuss.elastic.co/t/these-old-indices-should-be-deleted-by-es-according-to-ilm-policy-right/211189>\
**Category:** Elasticsearch\
**Tags:** ilm-index-lifecycle-management\
**Created:** [December 9, 2019, 8:31pm UTC](https://discuss.elastic.co/t/these-old-indices-should-be-deleted-by-es-according-to-ilm-policy-right/211189 "2019-12-09T20:31:41Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![EricJohnson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ericjohnson/32/53592_2.png) [@EricJohnson](https://discuss.elastic.co/u/EricJohnson)\
**Post date:** [December 9, 2019, 8:31pm UTC](https://discuss.elastic.co/t/these-old-indices-should-be-deleted-by-es-according-to-ilm-policy-right/211189/1 "2019-12-09T20:31:41Z")

</div>

Greetings.

I have implemented index lifecycle management on beats. Here's the configuration we are using:

```
PUT /_ilm/policy/beat_default_lifecycle_policy
{
    "policy": {
        "phases": {
        "hot": {
            "min_age": "0ms",
            "actions": {
                "rollover": {
                    "max_size": "5gb"
                },
                "set_priority": {
                    "priority": 100
                }
            }
        },
        "delete": {
            "min_age": "7d",
            "actions": {
                "delete": {}
            }
        }
    }
}
}

```

So... If an index is over 5GB it should be deleted in 7 days. The indexes are not, however, being deleted.

One index this ILM should apply to is _packetbeat-7.3.1-2019.09.10-000001_. Here are some of the settings;

```
{
  "index.blocks.read_only_allow_delete": "false",
  "index.priority": "1",
  "index.write.wait_for_active_shards": "1",
  "index.lifecycle.name": "beat_default_lifecycle_policy",
  "index.lifecycle.rollover_alias": "packetbeat-7.3.1",
  "index.mapping.total_fields.limit": "10000",
...
}

```

 ![index](https://us1.discourse-cdn.com/elastic/original/3X/7/a/7a7bd0f7ed2d2c5d5eb098c7d45d497fd8ba4637.png)

It's well over 5GB. Shouldn't this index be deleted by ES?

Thank you.

---

<div class="post-metadata">

**Author:** ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)\
**Post date:** [December 9, 2019, 10:21pm UTC](https://discuss.elastic.co/t/these-old-indices-should-be-deleted-by-es-according-to-ilm-policy-right/211189/2 "2019-12-09T22:21:04Z")

</div>

The size for rollover is the size of the primary, I don't know whether the screenshot's "Storage size" is just the primary or whether it factors in the size of the replica as well.

You can check with:

```auto
GET /_cat/shards/packetbeat-7.3.1-2019.09.10-000001?v

```

You can also see ILM explanation in:

```auto
GET /packetbeat-7.3.1-2019.09.10-000001/_ilm/explain?human

```

Which would be helpful to see.

---

<div class="post-metadata">

**Author:** ![EricJohnson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ericjohnson/32/53592_2.png) [@EricJohnson](https://discuss.elastic.co/u/EricJohnson)\
**Post date:** [December 9, 2019, 10:37pm UTC](https://discuss.elastic.co/t/these-old-indices-should-be-deleted-by-es-according-to-ilm-policy-right/211189/3 "2019-12-09T22:37:01Z")

</div>

Thank you for the reply. Interesting...

When I run that first command it outputs

> index shard prirep state docs store ip node  
> packetbeat-7.3.1-2019.09.10-000001 0 p STARTED 21088072 7.9gb 127.0.0.1 serverName  
> packetbeat-7.3.1-2019.09.10-000001 0 r UNASSIGNED

---

<div class="post-metadata">

**Author:** ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)\
**Post date:** [December 9, 2019, 10:49pm UTC](https://discuss.elastic.co/t/these-old-indices-should-be-deleted-by-es-according-to-ilm-policy-right/211189/4 "2019-12-09T22:49:57Z")

</div>

Okay it does look like it should be large enough to roll over.

What was the output of the second command?

---

<div class="post-metadata">

**Author:** ![EricJohnson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ericjohnson/32/53592_2.png) [@EricJohnson](https://discuss.elastic.co/u/EricJohnson)\
**Post date:** [December 9, 2019, 10:52pm UTC](https://discuss.elastic.co/t/these-old-indices-should-be-deleted-by-es-according-to-ilm-policy-right/211189/5 "2019-12-09T22:52:25Z")

</div>

That one shows an issue - but I'm not sure how to address it or if it's causing the problem.

> {  
> "indices" : {  
> "packetbeat-7.3.1-2019.09.10-000001" : {  
> "index" : "packetbeat-7.3.1-2019.09.10-000001",  
> "managed" : true,  
> "policy" : "beat\_default\_lifecycle\_policy",  
> "lifecycle\_date" : "2019-09-10T17:38:41.521Z",  
> "lifecycle\_date\_millis" : 1568137121521,  
> "age" : "90.21d",  
> "phase" : "hot",  
> "phase\_time" : "2019-09-10T17:38:41.609Z",  
> "phase\_time\_millis" : 1568137121609,  
> "action" : "rollover",  
> "action\_time" : "2019-09-10T17:40:02.228Z",  
> "action\_time\_millis" : 1568137202228,  
> "step" : "ERROR",  
> "step\_time" : "2019-09-26T02:40:08.666Z",  
> "step\_time\_millis" : 1569465608666,  
> "failed\_step" : "check-rollover-ready",  
> "step\_info" : {  
> "type" : "master\_not\_discovered\_exception",  
> "reason" : null,  
> "stack\_trace" : """MasterNotDiscoveredException[null]  
> at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.onTimeout(TransportMasterNodeAction.java:251)  
> at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325)  
> at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252)  
> at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:572)  
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688)  
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)  
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)  
> at java.base/java.lang.Thread.run(Thread.java:835)  
> """  
> },  
> "phase\_execution" : {  
> "policy" : "packetbeat-7.3.1",  
> "phase\_definition" : {  
> "min\_age" : "0ms",  
> "actions" : {  
> "rollover" : {  
> "max\_size" : "50gb",  
> "max\_age" : "30d"  
> }  
> }  
> },  
> "version" : 1,  
> "modified\_date" : "2019-09-10T17:38:41.280Z",  
> "modified\_date\_in\_millis" : 1568137121280  
> }  
> }  
> }  
> }

---

<div class="post-metadata">

**Author:** ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)\
**Post date:** [December 9, 2019, 10:54pm UTC](https://discuss.elastic.co/t/these-old-indices-should-be-deleted-by-es-according-to-ilm-policy-right/211189/6 "2019-12-09T22:54:10Z")

</div>

You should be able to retry this with:

```auto
POST /packetbeat-7.3.1-2019.09.10-000001/_ilm/retry

```

This should retry the step since it's in an error state.

In later versions of ES we've added automatic retry for some steps.

---

<div class="post-metadata">

**Author:** ![EricJohnson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ericjohnson/32/53592_2.png) [@EricJohnson](https://discuss.elastic.co/u/EricJohnson)\
**Post date:** [December 9, 2019, 10:57pm UTC](https://discuss.elastic.co/t/these-old-indices-should-be-deleted-by-es-according-to-ilm-policy-right/211189/7 "2019-12-09T22:57:03Z")

</div>

That operation was successful. Thank you.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 6, 2020, 10:57pm UTC](https://discuss.elastic.co/t/these-old-indices-should-be-deleted-by-es-according-to-ilm-policy-right/211189/8 "2020-01-06T22:57:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.