Using ILM rollover with multiple indices

Enviroment

ES: 7.8.1
Kibana: 7.8.1
Run in docker: 19.03.8, build afacb8b
Using docker image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1

Hi everybody,
I'm having a hard time with using ILM rollover with multiple indices.
For example, I have tree indices.

  • index-xxx
  • index-yyy
  • index-zzz
    and they have same alias: index-all

The index template is:

  • index pattern: index-*
  • Setting
{
  "index": {
    "lifecycle": {
      "name": "index-ilm",
      "rollover_alias": "index-all"
    }
  }
}
  • Alias
{
  "index-all": {}
}

The ILM rollover is enabled.
The problem is, when the ILM start, it tells me that
illegal_argument_exception: source alias [index-all] does not point to a write index
I read the doc and the doc tells me that I should set is_write_index: true for one of my index.
But I have multiple indices and ES only allow me to set one of my index, how can I deal with it.

Purpose

I want all my index can use rollover using same alias like:

  • index-xxx-00001
  • index-yyy-00001
  • index-zzz -00001
  • index-xxx-00002
  • index-yyy-00002
  • index-zzz -00002
  • shrink-index-xxx
  • shrink-index-yyy-00001
  • shrink-index-zzz -00001

Another Problem

When I disable the rollover , it follow each stage min_age, but when a index go into shrink/cold stage(shrink-index-zzz), I try to insert a new doc in index shrink-index-zzz, it will not auto create a new index and return 403 Forbiddent.

Thanks for helping!

If you want to use rollover I believe you need a policy per index name patter as they need to have different write aliases.

yearh... I agree. Is there have some better solution?

Why not use standard time based indices, but change from daily to weekly or monthly. If an index gets too large you can always split it later.

That is a good way to do it but I still want to use the rollover function. :slight_smile:

Then i believe you need to find a way to create/manage a policy per index pattern. :slight_smile: