Unable to automatize rollover using "index templates"

Elastic Stack Elasticsearch
1

Hi,

The indices of my ES cluster are structured as follows:

deviceA-2018    (size 5gb)
deviceA-2019    (size 1gb)
deviceB-2019    (size 100gb)

The deviceIDs are dynamic and unknown a priori and the indices sizes vary too much between them.

For automatization of index creation I am using "index templates" (e.g. set number of shards). This mechanism is very convenient because the indexation is made via Apache Storm and I do not need to take care of the existance of the index; if it does not exists, it is created with the template settings and if exists, the data is indexed directly.

PUT _template/device_template
{
  "index_patterns": ["*-*"],
  "settings": {
    "number_of_shards": 5
  },
  ...
}

For performance reasons I am restricting the max size of an index using "rollover" functionality.

PUT _ilm/policy/device_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "30GB"
          }
        }
      }
    }
  }
}
PUT _template/deviceA_template
{
  "index_patterns": ["deviceA-2019-*"],
  "settings": {
    "number_of_shards": 5
  },
  "settings": {
    "index.lifecycle.name": "device_policy",
    "index.lifecycle.rollover_alias": "deviceA-2019"
  }
  ...
}
PUT deviceA-2019-000001
{
  "aliases": {
    "deviceA-2019": {
      "is_write_index": true
    }
  }
}

My problem is that I cannot use these two features in conjuction in an automatic way. The configuration that I tried to use but does not work is:

PUT _template/device_template
{
  "index_patterns": ["*-*-000001"],
  "settings": {
    "number_of_shards": 5
  },
  "settings": {
    "index.lifecycle.name": "device_policy",
    "index.lifecycle.rollover_alias": "{indexWITHOUTrolloverID}" //e.g. deviceA-2019
  },
 "aliases": {
    "{indexWITHOUTrolloverID}": { //e.g. deviceA-2019
      "is_write_index": true
    }
  }
  ...
}

PUT _template/device_template
{
  "index_patterns": ["*-*-!00001"],
  "settings": {
    "number_of_shards": 5
  },
  "settings": {
    "index.lifecycle.name": "device_policy",
    "index.lifecycle.rollover_alias": "{indexWITHOUTrolloverID}" //e.g. deviceA-2019
  },
  ...
}

The problem that appears in this issue is very similar to my problem (https://github.com/elastic/kibana/issues/27967). Notice that this issue appears in kibana repo but it it is not related with kibana but with elasticsearch.

Is there any way to do delegate the creation of a index on demand by ElasticSearch with rollover enabled?

Thank you for your help!

2

Hi @pispo,

I am afraid not at present. Setting up the alias and index for use with ilm is a bit cumbersome as described in: https://github.com/elastic/elasticsearch/issues/42245.

You have to setup the first index and alias manually (or in your integration with Elasticsearch) for now.