Filebeat indices get grouped by ILM

Elastic Stack Beats
,
1

Hi,
my Problem is that Filebeat indices are not named like the default should be (on a daily basis), but the naming seems to be affected by the used Index Lifecycle Policy. So when they should stay in hot phase for 6 days and then should be deleted (which should be much easier to do, like setting a single number in Kibana), then all indices in this 6-day period are named like filebeat-7.3.2-2019.11.29-000009, which looks more like a rollup index name. The name seems to be given on the first day of a six-day period and then changes. If the hot phase would end after 30 days than only a single index per month would be created. The older data also ends with 000009. I have no idea where this comes from as in a previous test the index names displayed the correct date and without the 000009 suffix. Of interest in the settings (as inspected with Kibana) may be

"provided_name": "<filebeat-7.3.2-{now/d}-000009>"

although i nowhere set it like this. The index is then deleted on the 7th day, which also includes the newest data in the index which is only 2 days old at that point in time.

Hope someone can help, thanks!

2

Welcome! The index pattern you're seeing does look like a result of ILM triggering a rollover... have you configured a rollover policy? (You can check in Kibana). It's hard to say more without seeing the configuration, but if you share your Filebeat configuration and index / rollover details we could probably help further.

3

Thanks for your reply. In the filebeat.yml no specific index pattern is set, so I assume the default "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}" is used. As for a lifecycle policy I use

{
"policy": {
    "phases": {
        "hot": {
            "min_age": "0ms",
            "actions": {
                "rollover": {
                    "max_age": "6d",
                    "max_size": "50gb"
                }
            }
        },
        "delete": {
            "min_age": "1d",
            "actions": {
                "delete": {}
            }
        }
    }
 }
}

The Rollup-Job is currently stopped but index pattern was set to filebeat-7.3.2-*. Perhaps the rollup index name filebeat-7.3.2 was the problem...

4

Hi Stephan!

I'm also trying to figure out how to delete documents older than x days using ILM.
To do that, you need to base the index name on the document timestamp, just like pre-ILM date-style based naming (indexname-%Y.%m.%d etc). If using ILM rollover or some other naming, there's no guarantee that documents from other dates exists in the index that will be deleted by ILM after x days.

Please note how the Kibana ILM editor changes its text when having rollover enabled.
The policy.phases.delete.min_age means from index creation with rollover disabled and index rollover with rollover enabled.

So your findings make perfect sense with rollover on the 6th day and deletion on the 7th (1 day after rollover).
Just beware that your use of max_size can also trigger a rollover before that, so if you get 50 gb of data every hour on one day, there will be 24 indicies for day 1 and they will all be deleted after day 2.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.