# ILM keep rolling over empty indexes

**URL:** <https://discuss.elastic.co/t/ilm-keep-rolling-over-empty-indexes/332480>\
**Category:** Elasticsearch\
**Tags:** ilm-index-lifecycle-management\
**Created:** [May 4, 2023, 1:20am UTC](https://discuss.elastic.co/t/ilm-keep-rolling-over-empty-indexes/332480 "2023-05-04T01:20:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Adam\_Lin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_lin/32/99542_2.png) [@Adam\_Lin](https://discuss.elastic.co/u/Adam_Lin)\
**Post date:** [May 4, 2023, 1:20am UTC](https://discuss.elastic.co/t/ilm-keep-rolling-over-empty-indexes/332480/1 "2023-05-04T01:20:40Z")

</div>

Hi

according to the following release note, since 8.5.3, ILM won't rollover the empty index by default.

> <https://github.com/elastic/elasticsearch/pull/89557>
>
> Closes #86203, related to #89283
> 
> ILM will no longer roll over empty indices b…y default. This is accomplished by injecting a \`min\_docs: 1\` condition into the rollover dry run request in \`WaitForRolloverReadyStep\`. 
> 
> Such a condition will only be injected if there's not already a \`min\_docs\` (or \`min\_primary\_shard\_docs\`) condition associated with the rollover, so a policy can opt out of the new behavior by adding an explicit \`min\_docs: 0\` condition to the rollover action.
> 
> This can also be disabled on a cluster-wide basis by setting \`indices.lifecycle.rollover.only\_if\_has\_documents\` to \`false\`.

I'm using the elk server v8.6.0, and with the ILM policy

```auto
{
  "testpolicy" : {
    "version" : 2,
    "modified_date" : "2023-05-02T06:46:02.069Z",
    "policy" : {
      "phases" : {
        "warm" : {
          "min_age" : "1h",
          "actions" : {
            "shrink" : {
              "number_of_shards" : 1
            },
            "forcemerge" : {
              "max_num_segments" : 1
            }
          }
        },
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_primary_shard_size" : "50gb",
              "max_age" : "1h",
              "min_docs" : 1
            }
          }
        },
        "delete" : {
          "min_age" : "3h",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      },
      "_meta" : {
        "managed" : true,
        "description" : "Test ILM policy"
      }
    },
    "in_use_by" : {
      "indices" : [
        ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.01-2023.05.02-000002",
        ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.03-000022",
        ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.04-000028",
        ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.02-2023.05.03-000040",
        ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.03-000026",
        ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.03-000024"
      ],
      "data_streams" : [
        "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03",
        "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.02",
        "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.01"
      ],
      "composable_templates" : [
        "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======"
      ]
    }
  }
}

```

after few days, the empty indexes still stay there.

```auto
green open .ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.01-2023.05.02-000002 wJEpsDXEQqaJ-dRPbu4hwg 1 0 0 0 225b 225b
green open .ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.03-000022 GWNx3vOhSj-G74SxG7OCcQ 1 0 561 0 446.3kb 446.3kb
green open .ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.04-000028 Zxc6LYWhRGanBl5F3lSvNw 1 0 399 0 416.1kb 416.1kb
green open .ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.02-2023.05.03-000040 mALAWq6wTMuqC8C769MOsg 1 0 0 0 225b 225b
green open .ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.03-000026 K8QRwp39SuuJFK8SJzksrg 1 0 561 0 534.4kb 534.4kb
green open .ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.03-000024 gg_9GVgWS1WXckdKVQUW8w 1 0 564 0 449.1kb 449.1kb

```

I wish that the age-out will be going to the warm phase after an hour and deleted after 3 hours.  
There is no problem with ILM within one day, and it scrolls normally, but on the next day, a new index is created, and the old index stays in the HOT stage

for example, the index ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.01-2023.05.02-000002" is already age-out, but when I check the stat this index, it still stops in the hot phase.

```auto
{
  "indices" : {
    ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.01-2023.05.02-000002" : {
      "index" : ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.01-2023.05.02-000002",
      "managed" : true,
      "policy" : "testpolicy",
      "index_creation_date_millis" : 1683013920597,
      "time_since_index_creation" : "1.72d",
      "lifecycle_date_millis" : 1683013920597,
      "age" : "1.72d",
      "phase" : "hot",
      "phase_time_millis" : 1683013922027,
      "action" : "rollover",
      "action_time_millis" : 1683013922443,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1683013922443,
      "phase_execution" : {
        "policy" : "testpolicy",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_primary_shard_size" : "50gb",
              "max_age" : "1h",
              "min_docs" : 1
            }
          }
        },
        "version" : 2,
        "modified_date_in_millis" : 1683009962069
      }
    }
  }
}

```

this is my indexes and datastreams:

```auto
{
  "index_templates" : [
    {
      "name" : "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======",
      "index_template" : {
        "index_patterns" : [
          "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-*"
        ],
        "template" : {
          "settings" : {
            "index" : {
              "lifecycle" : {
                "name" : "testpolicy",
                "rollover_alias" : "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======"
              },
              "mapping" : {
                "total_fields" : {
                  "limit" : "10000"
                }
              },
              "refresh_interval" : "5s",
              "number_of_shards" : "1",
              "max_docvalue_fields_search" : "200",
              "query" : {
                "default_field" : [
				   ...
                ]
              },
              "number_of_replicas" : "0"
            }
          },
          "mappings" : {
		   ...
          }
        },
        "composed_of" : [],
        "priority" : 150,
        "data_stream" : {
          "hidden" : false,
          "allow_custom_routing" : false
        }
      }
    }
  ]
}

```

```auto
{
  "data_streams" : [
    {
      "name" : "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.01",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.01-2023.05.02-000002",
          "index_uuid" : "wJEpsDXEQqaJ-dRPbu4hwg"
        }
      ],
      "generation" : 3,
      "status" : "GREEN",
      "template" : "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======",
      "ilm_policy" : "testpolicy",
      "hidden" : false,
      "system" : false,
      "allow_custom_routing" : false,
      "replicated" : false
    },
    {
      "name" : "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.02",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.02-2023.05.03-000040",
          "index_uuid" : "mALAWq6wTMuqC8C769MOsg"
        }
      ],
      "generation" : 43,
      "status" : "GREEN",
      "template" : "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======",
      "ilm_policy" : "testpolicy",
      "hidden" : false,
      "system" : false,
      "allow_custom_routing" : false,
      "replicated" : false
    },
    {
      "name" : "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.03-000026",
          "index_uuid" : "K8QRwp39SuuJFK8SJzksrg"
        },
        {
          "index_name" : ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.04-000028",
          "index_uuid" : "Zxc6LYWhRGanBl5F3lSvNw"
        },
        {
          "index_name" : ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.04-000030",
          "index_uuid" : "fAxgWI5nRkKXJnCqMHhY3g"
        },
        {
          "index_name" : ".ds-ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======-2023.05.03-2023.05.04-000032",
          "index_uuid" : "8V2WJXDeT4CU7OzZZ8gNYw"
        }
      ],
      "generation" : 32,
      "status" : "GREEN",
      "template" : "ovuwyyjnozuwgljvfyys4mbngiys2mrvgaxdqmikbi======",
      "ilm_policy" : "testpolicy",
      "hidden" : false,
      "system" : false,
      "allow_custom_routing" : false,
      "replicated" : false
    }
  ]
}

```

Could someone help to correct me if there have configuration errors or any stuff I misunderstood?

Thanks.

---

<div class="post-metadata">

**Author:** ![Adam\_Lin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_lin/32/99542_2.png) [@Adam\_Lin](https://discuss.elastic.co/u/Adam_Lin)\
**Post date:** [May 5, 2023, 1:16am UTC](https://discuss.elastic.co/t/ilm-keep-rolling-over-empty-indexes/332480/2 "2023-05-05T01:16:45Z")

</div>

Hello

anyone could help?

---

<div class="post-metadata">

**Author:** ![Adam\_Lin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_lin/32/99542_2.png) [@Adam\_Lin](https://discuss.elastic.co/u/Adam_Lin)\
**Post date:** [May 8, 2023, 7:28am UTC](https://discuss.elastic.co/t/ilm-keep-rolling-over-empty-indexes/332480/3 "2023-05-08T07:28:20Z")

</div>

Could anyone help for this?

---

<div class="post-metadata">

**Author:** ![Tyler\_Perkins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tyler_perkins/32/118993_2.png) [@Tyler\_Perkins](https://discuss.elastic.co/u/Tyler_Perkins)\
**Post date:** [May 26, 2023, 1:52pm UTC](https://discuss.elastic.co/t/ilm-keep-rolling-over-empty-indexes/332480/4 "2023-05-26T13:52:39Z")

</div>

Hi Adam,  
Sorry you're having trouble with ILM. With that feature in 8.5 we are trying to prevent empty indices which have overhead in the cluster and no value.

If I understand correctly, you want to use this feature and avoid empty indices, but you still want to rollover on your expected schedule. If so, I would suggest to try removing the "min\_docs" : 1 from the Hot phase in your ILM policy and see if you get the expected behavior. I think the distinction here is between 'rollover' and 'index creation or not'. I think your usage of min\_docs is preventing ILM to move to the next phase if there isn't at least one doc, whereas without that parameter it can move on even if there were no docs (but there will not be an empty index).

If [the documentation for this](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-rollover.html#ilm-rollover-options) is unclear please let us know.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 23, 2023, 1:53pm UTC](https://discuss.elastic.co/t/ilm-keep-rolling-over-empty-indexes/332480/5 "2023-06-23T13:53:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.