Sanity check on ILM

Elastic Stack Elasticsearch
#1

I have spent the last 3 weeks off and on trying to get ILM to work. I have datastreams going but one client I am using (syslog-ng) does not yet know about datastreams and uses index in the _bulk rather than create.

I found the documentation tantalizingly incomplete. It seems that the authors made assumption about what was obvious that were clearly wrong in my case. Common problem with technical documentation.

The curicial bit of information that I lacked was the link between the rollover alias and the alias set on the initial creation of the index with "is_write_index": true and that this is what you use to write to the index.

Assuming that I have this all right! It seems to be working as expected.

So here is my howto for an sanity check and for anyone else as stupid as me who tries to set up ILM from scratch. I found this this link most useful but it still requires some joining of dots...

  1. create your index and bootstrap the write/rollover alias:
curl -X PUT "localhost:9200/timeseries-000001?pretty" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    “timeseries-write": {
      "is_write_index": true
    }
  }
}

I could not find any way of doing this in kibana but the following steps can be...

  1. create the lifecycle policy (say timeseries-policy)
  2. create the index template with these settings 
{
  "index": {
    "lifecycle": {
      "name": “timeseries-policy",
      "rollover_alias": “timeseries-write"
    }
  }
}
  1. start writing to timeseries-write

Improvement or corrections solitited

#2

Thanks for using Elasticsearch. I believe the steps you highlighted are correct, with the only note to execute step 3 (creating the index template) before step 1 (creating the index with the alias and is_write_indexconfiguration) in order for the index.lifecycle.name and index.lifecycle.rollover_alias settings to be configured automatically when you create the index (executing the steps in the order you posted would render the timeseries-000001 index as unmanaged by ILM as the said index.lifecycle... settings are not configured for the index, but only for future indices that match the template index pattern).
Otherwise, with the steps order you provided, there needs to be an extra step to associate the timeseries-policy and the rollover alias with the timeseries-000001 index using the update indices settings API.

The order of steps should be:

  1. create index lifecycle policy
  2. create index template that configures the lifecycle.name and lifecycle.rollover_alias settings
  3. create the index with the alias configuration

We recognised some of the challenges associated with alias management and the is_write_index configuration and we developed data streams which I see you are already using for other indices.

1 Like
#3

with regard to order: absolutely! of course you need to create the template first! My bad.

I tried to edit the original to fix this but I got a permission error when I tried to save.

R

1 Like