How to archive data from a Transform?

Hi,

A Transform add data to a unique index.
This index is getting too big.
I would like to store the data (more than 2 days old) to another warm index.

How could I achieve that?
Thanks a lot for your time!

You will want to look at ILM to manage that for you.

I try the alias + rollover.
The index are well created, and can change their lifecycle status, that's good...

But there is a huge problem with data integrity.

Because if the index is "n" minutes old, it will be archive and a new one will be create, but a Transform can update the recent data, so quite often the data is duplicated between two indexes (0003 and 0004 for example).

How can I set something as a rollover only on the "old data" ? The 10 most recent minutes should stay active.

This is the same ID but on different indexes, maybe I can delete the old ones, but how?

image

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.