Rollover for Daily indices in Elasticsearch

Elastic Stack Elasticsearch
1

Hello,
I am trying to implement rollover mechanism to my environments. Let me explain as is and to be design;
Log data stores in syslog-%{+YYYY.MM.dd} indices right now. I keep data for 3 days. No rollover - No warm or cold phase. It is going to delete phase directly. So this is the as is design and I have
syslog-2023-01-29
syslog-2023-01-30
syslog-2023-01-31

And I made some changes our ilm policy and apply rollover settings. It is running but there is a point;
I need to separate index for everyday. So I added these ILM pattern settings in my logstash conf shown as below

            ilm_rollover_alias => "syslog"
            ilm_pattern => "{now/d}-000001"
            ilm_policy => "timeseries-syslog"

Result and Question;
syslog-2023-02-01-000001 => it is ok
syslog-2023-02-01-000002 => Why this pattern starts from 000002 ? How can I provide to start from 000001 when every new index created ? Is it normal behavior ?

Thank You

2

The whole point of rollover is to allow new indices to be created based on age or size. This allows indices to cover a longer time period when data volumes are low and roll over more frequently during peak hours. This means that index creation will not align with specific points in time, e.g. change of date. These indices all have a suffix that will increment, which ensures a unique index name.

If you want to use rollover I would recommend not trying to control naming convention or exactly when rollover is performed. ILM will handle the lifecycle based on index metadata and not the name.

If you want indices to correspond to specific time periods, e.g. exactly a day per index, then do not use rollover and instead set the index name based on the event time as was commonly done before rollover became available.

1 Like
3

Thank You very much, Then I will merge them to one syslog index and I will use rollover.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.