Simple question about Index Rollover

Elastic Stack Kibana
1

I have a simple policy for index rollover every day (or every 1gb). I use an index template -> index pattern->index alias (see bellow).

However it does not seem to rotate. Any ideas why (or how to test it? E.g if I send _rollover to the index_alias will that force the rotation even if the requirements are not met?)

Here is how I created it:

  1. I first pointed logstash to the alias: index_from_logstash_test_alias
  2. I created the index template: index_template (with index pattern index_from_logstash_test_alias* )
  3. Bootstrap index with
    Bootstrap index:
PUT index_from_logstash_test_alias-000001
{
  "aliases": {
    "index_from_logstash_test_alias": {
      "is_write_index": true
    }
  }
}

Some more info if needed:


PUT _index_template/test_template
{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "testindexpolicy",
          "rollover_alias": "index_from_logstash_test_alias"
        },
        "number_of_replicas": "0"
      }
    },
    "mappings": {
      "properties": {
        "@version": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "@timestamp": {
          "type": "date"
        },
        "app_server": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "app_server2": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    },
    "aliases": {
      "test-search-all": {}
    }
  },
  "index_patterns": [
    "index_from_logstash_test_alias*"
  ]
}

And the policy:

{
  "testindexpolicy" : {
    "version" : 1,
    "modified_date" : "2023-03-17T12:08:04.639Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "set_priority" : {
              "priority" : 100
            },
            "rollover" : {
              "max_primary_shard_size" : "1gb",
              "max_age" : "1d"
            }
          }
        }
      }
    },
    "in_use_by" : {
      "indices" : [ ],
      "data_streams" : [ ],
      "composable_templates" : [
        "test_template"
      ]
    }
5

What is the output of

GET index_from_logstash_test_alias-000001/_ilm/explain

The other thing I see ... not sure if it is a typo

In the template

        "lifecycle": {
          "name": "Testindexpolicy", <!--- Capitialized

And the policy:

{
  "testindexpolicy" : { <!--- Not Capitalized
6

yep that was a typo...Thanks - I corrected it so not to confuse others...

But the _ilm/explain command returned something interesting: "managed" : false"

So I double-checked the template to make sure that "rollover_alias" is pointing towards the correct alias. And is indeed index_from_logstash_test_alias. Also the index pattern seems ok... Maybe I am missing something ?

Here is also the output from GET /_alias/index_from_logstash_test_alias:

{
  "index_from_logstash_test_alias-000001" : {
    "aliases" : {
      "index_from_logstash_test_alias" : {
        "is_write_index" : true
      }
    }
  }
}
7

I am new to this Elastic thing so I could be wrong...

I think you should start over and deploy stuff in this order:

  1. The ILM policy
  2. The Index Template (with the index pattern and a reference to the policy)
  3. The index itself (with the bootstrapping or maybe not)

If you are lucky, the system will see that your template has an index pattern that matches your new index, pick it, and use the referenced policy. If you are not so lucky, try adding a ridiculous high priority (999) to your template so it has a better chance of getting matched and picked.

8

change to

  "index_patterns": [
    "index_from_logstash_test_alias-*"
  ]

then try to rollover your alias

POST index_from_logstash_test_alias/_rollover

and then look at _ilm/explain ... and post the full output not just a single line.

1 Like
9

Good idea Stephen, thanks. However I already had (in another cluster) a similar index rotation policy that worked fine with "index_from_logstash_test_alias*"

So I tried this: I directly tried
POST index_from_logstash_test_alias/_rollover and it worked!
So after this _ilm_explain indicated that a new index is created and is managed by the policy index_from_logstash_test_alias-000002
(yet index_from_logstash_test_alias-000001 is still not managed). So I guess the next rollover will occur automatically after one day.

I am not sure what was the problem - maybe bootstraping the first index should have been done via the template

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.