How to combine transform with rollover?

Elastic Stack Elasticsearch
1

Hello,

I was not able to find any information if it is possible to rollover transform destination index? I tried and that did not work, but maybe I am missing something. Currently I set destination index to "foo-00001" and attach policy to it. Then foo-00002 is created, but transform writes to foo-00001 anyway. So is it possible to

My goal is to aggregate data depending on how old it is: for last 2 weeks I want to have date histogram for each hour, for last 3 months - date histogram for each day and so on.

2

For rollover its best to use aliases, so instead of writing to a concrete index you configure transform to write to an index alias. You can than use rollover on the alias. I think that's the best possible option using the built-in functionality.

Just mentioning another option: Transform has _update API you can use to switch the destination to another index (e.g. you can use _update to migrate your existing transforms to use index alias destinations).

3

Thank you for reply.

What I tried now:

  1. Created index template "metrics"
  2. Created ILM policy
  3. Attached policy to index template with alias "metrics-alias"
  4. Via transform update API changed transform dest.index to "metrics-alias"
  5. Tried starting transform

But got error:
error: "{"msg":"[runtime_exception] Could not create destination index [metrics-alias] for transform [metrics]","path":"/_transform/metrics/_start","query":{},"statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"runtime_exception\",\"reason\":\"Could not create destination index [metrics-alias] for transform [metrics]\"}],\"type\":\"runtime_exception\",\"reason\":\"Could not create destination index [metrics-alias] for transform [metrics]\",\"caused_by\":{\"type\":\"illegal_state_exception\",\"reason\":\"index and alias names need to be unique, but the following duplicates were found [metrics-alias (alias of [metrics-alias/OzVFL0H-TCGRGXlOdJVoNw])]\"}},\"status\":500}"}"

Could you guide me what am I doing wrong?

4

Thanks, this sounds like a bug to me. Will try to reproduce.

5

I tried to reproduce it, but wasn't able. Transform checks on start as well as on update whether an index exists or not. It somehow seems that the destination index got created while update was running.

Otherwise I see no problem with your approach. If you are using index templates and aliases you should always create the destination yourself.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.