Index not rollover when the date changes

Elastic Stack Elasticsearch
1

Hi Team

With ILM policy, we can do rollover using different parameters such as size/document_count/age. This works fine and if we create first index with date-math, we are able to generate date-wise index.

But the problem is the index rollover happens only when the rollover conditions are satisfied, it does not rollover when the date is changed but rollover conditions are not satisfied.

So for example, lets consider that we have created index with date-math and rollover settings are such that index will rollover on 50 gb size. Now we have some indices which are getting rollover at every 50 gb. e.g., index-name-2021.04.29-000035, index-name-2021.04.29-000036 and so on...

Lets consider that index-name-2021.04.29-000036 is created at 23.57 on date 04/29/2021 and by the time date is changed to 04/30/2021, it may not have 50 gb data, so it won't rollover evenif the date is changed. So, whatever the new incoming data is there on 04/30/2021 will be indexed to index-name-2021.04.29-000036 until it reaches to 50 gb and after that only new index will be created with index-name-2021.04.30-000037. So some of the data from 04/30/2021 are indexed inside index-name-2021.04.29-000036.

Ideally this index should be rollover as and when the date changes and evenif the size condition is not satisfied. Is there a way that we can achieve this ?

Any help will be much appreciated.

Thanks & Regards

2

Rollover works on size and age and does not consider date math, so I do not think you can achieve what you are looking to do. When using rollover it is therefore recommended to not try to align the indices to strict time boundaries. Why does the indices have to cut off at a date boundary in the first place?

3

Christian is correct. The approach ILM takes is different from previous date-based indices. The date in the index name under ILM is the date that the index policy first created its index.

4

Thanks @Christian_Dahlqvist and @warkolm
So that means even-if we have applied date-math for first index, there can be other date data available for the given index.
So, for example, rollover index is created at 04/29, 23.57(11.57 pm) with name index-name-2021.04.29-000035 will contain data from next day 04/30 until the former index reaches the rollover condition.

5

Thanks for using Elasticsearch and ILM.

Daily/weekly indices creation could be achieved using logstash.

You could combine that with ILM for the other purposes, just not time based rollovers (eg. data migration, shrinking, mounting as searchable snapshot, deletion).