Hi everyone,
My use case is the following : I have continuously produced time-series data + one year history (both outside Elastic). I want to index them into Elastic in such a way that data is deleted after one year (according to the @timestamp field).
Data streams seem to be the perfect solution for the newly producted time-series data. They get indexed as soon as they are created, and the ILM will delete the associated backing indices at the right moment in one year.
However, I'm stuck with the historical datas. How to index them in such a way that the historical data will be deleted at the right time ? As the rollover is based on the index age and not the documents @timestamp fields, all associated backing indices will be also deleted in one year, even if they contains older data. In my use case, this typically means that the oldest historical data will remain two years in the cluster, which is not the expected behaviour.
Do you have any ideas to fix this ?