Hello, does anyone have successfully applied ILM on a transform or rollup index? or it's possible only with normal index? thanks
Welcome to our community! 
Are you having problems implementing this?
yes i want to apply ILM on an sumuriezed part of my data for 2 months ( transfrom or rollup ) without duplicate data that is principale index in hot phase.
I too am interested in how to accomplish this. I want to age out old rollup data through the ILM stages.
i think it's not possible so i used transform api instead of rollup..
Ok. How did that work? Doesn't transform have the same problem that it is one named index and not a index pattern that is used to move the data through ILM?
ILM and Transform
For using ILM on the destination index, you have to create the destination upfront before starting the transform. Note: It is not sufficient to create an index template, but you have to create the index. It is possible to let transform write to an index alias, write aliases are supported, too. Transform will only create the destination index for you, if either an index or alias does not exist at the time you start the transform. You can use the preview API to see what transform suggests as index mappings.
Problem explained:
Transform updates documents in the destination, this can lead to side effects together with the usage of ILM. Transform will only write to the current destination, in case of a rollover this might create duplicates, because transform can not delete documents in the previous concrete index.
Possible Solution:
If you have time-based indices and your transform contains a group_by based on date_histogram you can attach an ingest pipeline to dest and configure a date index name processor with the timestamp field from the group by as input. This won't by design create duplicates. The idea is basically that by using the index name processor that you create daily/weekly/monthly indices. This isn't based on size, but maybe that's ok for you.
Addition:
If you use a recent version of the stack align_checkpoints which is part of settings - default true - allows you to only write complete buckets.
If and only if the time field used in sync and the timefield used for the date_histogram in group_by are the same the problem of duplicates in the destination due to index rollover is technically not possible. For this special case just create the destination youself and configure ILM for it.
If your sync field and the date_histogram field are not the same, this won't work reliable and I suggest the index name trick explained above.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.