Using ILM with mutable data and multiple data tiers

Hi,

We are experiencing load and cost issues related to mutable data.

This is our situation:

  • Every day we receive 30 million documents we need to index
  • 29.9 million of these documents are not queried often and can be stored in warm or colder storage
  • These documents get updated multiple times shortly after insertion
  • Sporadically, some can be updated days, if not months after the original insertion
  • We are only interested in the latest state of each document

We would like to store these 29.9 million mutable documents in a separate index, which can be gradually moved to colder storage to avoid costs.

Any idea on how we could do this?

Kind regards,
Andreas