Automate rollover for daily indices using ILM

[Akshay_Pawar]

Hello, I am looking into automating rollover using Index lifecycle management for daily indices.

I am looking for the pattern as follows so that I would be seamlessly able to search across all three indices

    rollover-test-2020.09.18 (pre-create template with this alias) 
        -rollover-test-2020.09.18-000001  (pre-create index)
        -rollover-test-2020.09.18-000002
        -rollover-test-2020.09.18-000003

    rollover-test-2020.09.19 (pre-create template with this alias)
        -rollover-test-2020.09.19-000001 (pre-create index)
        -rollover-test-2020.09.19-000002
        -rollover-test-2020.09.19-000003

As rollover API requires an alias, I have to pre-create indices so that I can use the "is_write_index" and "alias" setting for the original index. Also, I figured that I will have to pre-create template for every daily index so that it holds the rollover_alias (rollover-test-2020-09-19 in above case) settings and when ILM rolls over the index, the new index can apply the rollover alias from the template.

Is there any better solution for this where in I don't have to create template for every daily index?

[warkolm (Mark Walkom)]

Welcome to our community!

If you are using ILM, then there's no need create daily indices like that. It's supposed to handle that for you.

[Akshay_Pawar]

@warkolm, I am creating daily indices because its an requirement.

[warkolm (Mark Walkom)]

Ok, but you can tell ILM to do that directly by using a max_age of 24h.

ILM is not intended to me used in the way you are looking at.

[Akshay_Pawar]

The reason I am planning to use automated rollover for daily indices is to rollover the index when the index reaches ~25GB.
Eg:
When rollover-test-2020.09.18-000001 reaches ~25GB, rollover so that a new index will be created rollover-test-2020.09.18-000002 and shards would be distributed.

[warkolm (Mark Walkom)]

Yep I get that. The intent with ILM is that you also define the max_age to be 24h, thereby also getting the daily rollover.

With your other approach, there's no way in ILM do do that. You'd need to do the setup each day, as you are effectively deploying a new policy each day.