Manage multiple indicies with one ILM-Policy

[cwiechmann (Chris)]

Hi All,

I have the following structure of indices, that are created as needed, based on a template, to store documents received from a different regions:

some-data-from-region-us-000001
some-data-from-region-eu-000001
.....

The region ID is configurable, so I don't have it beforehand. The template is configured/linked to an ILM-Policy.
During index creation, each regional index gets it's own regional alias: (e.g. some-data-from-region-eu-alias), which is used to write data to this index.

That means, I have multiple Write-Indices (active) but each with a different regional aliases.
But this conflicts with the ILM-Feature, as I can configured only ONE alias. Hence I'm getting the following error:

index.lifecycle.rollover_alias [some-data-from-region] does not point to index [some-data-from-region-us-000001]

Does this mean, I have to maintain an ILM-Policy & Template for each regional index to use ILM? Or perhaps it's possible to update the indices rollover-alias after the index have been created. (e.g. using an interval, that validates the alias)

I'm asking, to make sure, that there is no other option I have overlooked.

I think it's related to this GitHub issue: https://github.com/elastic/elasticsearch/issues/38454, which has been closed in favor to Data-Streams. Unfortunately I cannot use Data Streams because upserts are not supported to my knowledge. I'm using Logstash to send documents.

If someone has some tips or ideas how to handle this use-case or just share best/practices/recommendations I would appreciate it.

Thanks!
Chris

[dakrone (Lee Hinman)]

Hi Chris, welcome!

To answer your question, if you are unable to use data streams for now, then yes, you'll have to maintain a separate template for each alias you're using (you don't need a separate ILM policy, they can all use the same policy).

You certainly could also update the index.lifecycle.rollover_alias setting dynamically after the index has been created (it's a dynamic setting and you can change it at any time). If you wanted to do that on an interval that would certainly work, requiring you to configure only a single template.

Thanks for letting us know why data streams won't work for now. In the future it'd be nice to be able to support your situation, and hopefully we'll be able to do that.

[cwiechmann (Chris)]

Thanks for the confirmation @dakrone!