Rollover Indexes Using ILM

[krish1 (Krish Godiawala)]

I am currently using ES version 7.17.0 and trying out rollover indexes using the ILM. I have read through the documentation and my use case is to rollover my ES index every Monday midnight once a week i.e, rollover by age. I notice that ILM indexes requires an index-pattern which is appended by an increasing number everytime an index is rolled over. I have a couple of questions about rollovers I couldn't completely capture from the document.

1. Is there a way to get a custom naming scheme for the rollover index? This because I'm looking to query a weeks worth of data and my query would move from index to index. For example is it possibly to get a naming scheme as my-index-2023-12-1, my-index-2023-12-2, my-index-2023-12-3, my-index-2023-12-4, my-index-2024-01-1. Alternatively is there a way to reverse lookup an index name based on the period/day it was rolled over?
2. If I search against the rollover_alias created as a part of the index template will that search against all the indexes(In the hot and warm phase)[similar to multi_get] or will it only search the is_write_index.

[Christian_Dahlqvist]

Before rollover was available time-based indices used a string in the index name to indicate the time period it covered and the process writing the data was responsible for setting the correct index name based on the timestamp in the data. This allowed users to align new index creation with neat daily, weekly or monthly patterns, just like you want to do. One drawback with this approach was that the shard size could vary a lot if ingest volume varied over time. This can have a big impact on performance, and it is generally desirable to have an even shard size for time-based indices.

Rollover was intrucuced to solve this issue and it allows you to roll over based on size or a max age. This allows you to achieve a much more even shard size, which is beneficial. What you give up as a tradeoff when using this is control over when indices are rolled over. This is for most use cases with time-based indices not an issue.

If you feel you need to cut over indices at specific times and that this is more important than achieving a more even shard size distribution I would recommend not using rollover and instead use the older approach. You can still use ILM for deleting indices and moving them through phases with indices created this way (you just do not specify that you use rollover and ILM will look at the index creation date).