We are creating an aggregated set of data using the rollup job feature in Elasticsearch 7.5.2. Creating the rollup job works and we have an index that contains the aggregated data, but we're using a hot/warm architecture and so want to rollover the rollup indexes so that we can move them to our warm nodes and then after a period of time we want to delete old data.
I've created a life cycle policy and a template that just defines an alias. I also created an index with the <alias_name>-<date>-00001 naming convention as described in the docs for setting up an ILM, but if I try to create a new rollup job and direct it at the index it says I can't because the index already exists. It also prevents me from just pointing at the alias. If I point it at an index that doesn't exist but follows the same naming convention, the index doesn't pick up on the template so I get an ILM error about a missing alias.
I'm sure I'm just missing something obvious.
Thanks for any help.
Hello @rhalliday
Unfortunately it is not possible to use ILM & Rollup features together.
The reasons are well explained on this Github issue: https://github.com/elastic/elasticsearch/issues/33065
One of the reasons is the fact at the moment Rollup search allows 1 rollup index (see limitations).
As you'll find in the linked issue, Rollups will be "built as an action" in ILM in the future (see https://github.com/elastic/elasticsearch/issues/48003).
Hi @Luca_Belluccini,
Thanks for your response.
That's a bit of a shame as it means our rollup index would end up being huge.
We'll have a look into data transforms and see if we can get the same effect with those.
Thanks,
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.