# Filebeat indices get grouped by ILM

**URL:** https://discuss.elastic.co/t/filebeat-indices-get-grouped-by-ilm/210144
**Category:** Beats
**Tags:** ilm-index-lifecycle-management, filebeat
**Created:** [December 2, 2019, 9:27am UTC](https://discuss.elastic.co/t/filebeat-indices-get-grouped-by-ilm/210144 "2019-12-02T09:27:00Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![stk1](https://avatars.discourse-cdn.com/v4/letter/s/ee59a6/32.png) [@stk1](https://discuss.elastic.co/u/stk1)
#### Post date: [December 2, 2019, 9:27am UTC](https://discuss.elastic.co/t/filebeat-indices-get-grouped-by-ilm/210144/1 "2019-12-02T09:27:00Z")

</div>

Hi,  
my Problem is that Filebeat indices are not named like the default should be (on a daily basis), but the naming seems to be affected by the used Index Lifecycle Policy. So when they should stay in hot phase for 6 days and then should be deleted (which should be much easier to do, like setting a single number in Kibana), then all indices in this 6-day period are named like filebeat-7.3.2-2019.11.29-000009, which looks more like a rollup index name. The name seems to be given on the first day of a six-day period and then changes. If the hot phase would end after 30 days than only a single index per month would be created. The older data also ends with 000009. I have no idea where this comes from as in a previous test the index names displayed the correct date and without the 000009 suffix. Of interest in the settings (as inspected with Kibana) may be

> "provided\_name": "\<filebeat-7.3.2-{now/d}-000009\>"

although i nowhere set it like this. The index is then deleted on the 7th day, which also includes the newest data in the index which is only 2 days old at that point in time.

Hope someone can help, thanks!

---

<div class="post-metadata">

### Author: ![faec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/faec/32/46988_2.png) [@faec](https://discuss.elastic.co/u/faec)
#### Post date: [December 6, 2019, 5:54pm UTC](https://discuss.elastic.co/t/filebeat-indices-get-grouped-by-ilm/210144/2 "2019-12-06T17:54:02Z")

</div>

Welcome! The index pattern you're seeing does look like a result of ILM [triggering a rollover](https://www.elastic.co/guide/en/elasticsearch/reference/current/using-policies-rollover.html)... have you configured a rollover policy? (You can [check in Kibana](https://www.elastic.co/guide/en/kibana/7.5/index-lifecycle-policies.html)). It's hard to say more without seeing the configuration, but if you share your Filebeat configuration and index / rollover details we could probably help further.

---

<div class="post-metadata">

### Author: ![stk1](https://avatars.discourse-cdn.com/v4/letter/s/ee59a6/32.png) [@stk1](https://discuss.elastic.co/u/stk1)
#### Post date: [December 10, 2019, 9:19am UTC](https://discuss.elastic.co/t/filebeat-indices-get-grouped-by-ilm/210144/3 "2019-12-10T09:19:41Z")

</div>

Thanks for your reply. In the filebeat.yml no specific index pattern is set, so I assume the default `"filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"` is used. As for a lifecycle policy I use

> ```auto
> 
> ```

```
{
"policy": {
    "phases": {
        "hot": {
            "min_age": "0ms",
            "actions": {
                "rollover": {
                    "max_age": "6d",
                    "max_size": "50gb"
                }
            }
        },
        "delete": {
            "min_age": "1d",
            "actions": {
                "delete": {}
            }
        }
    }
 }
} 

```

The Rollup-Job is currently stopped but index pattern was set to filebeat-7.3.2-\*. Perhaps the rollup index name filebeat-7.3.2 was the problem...

---

<div class="post-metadata">

### Author: ![anderssonjohan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anderssonjohan/32/60037_2.png) [@anderssonjohan](https://discuss.elastic.co/u/anderssonjohan)
#### Post date: [December 29, 2019, 2:16pm UTC](https://discuss.elastic.co/t/filebeat-indices-get-grouped-by-ilm/210144/4 "2019-12-29T14:16:36Z")

</div>

Hi Stephan!

I'm also trying to figure out how to delete **documents** older than x days using ILM.  
To do that, you need to base the index name on the document timestamp, just like pre-ILM date-style based naming (indexname-%Y.%m.%d etc). If using ILM rollover or some other naming, there's no guarantee that documents from other dates exists in the index that will be deleted by ILM after x days.

Please note how the Kibana ILM editor changes its text when having rollover enabled.  
The policy.phases.delete.min\_age means from index **creation** with rollover **disabled** and index **rollover** with rollover **enabled**.

So your findings make perfect sense with rollover on the 6th day and deletion on the 7th (1 day after rollover).  
Just beware that your use of max\_size can also trigger a rollover before that, so if you get 50 gb of data every hour on one day, there will be 24 indicies for day 1 and they will all be deleted after day 2.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 26, 2020, 2:16pm UTC](https://discuss.elastic.co/t/filebeat-indices-get-grouped-by-ilm/210144/5 "2020-01-26T14:16:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
