Index rollover issues

Elastic Stack Elasticsearch
1

Suppose I'm creating the ES index using beat system or logstash with date time interpolation.

  1. how do I configure rolling over of index if size reaches at certain GBs in a single day? I did the following but I'm doing something wrong.
    I created an ILM and then an index template and linked the ILM and let logstash create the new index. But index gives the following error. Any idea?
    illegal_argument_exception: index.lifecycle.rollover_alias [test-index] does not point to index [test-index-2021.12.01]

  2. What is the best practice? Create the index from beat/logstash with date or configuring entire rollover stuff from ES itself?

2

Welcome to our community! :smiley:

Please share your policy and index template.

1 Like
3

Thank you.

Here you go.

Index settings:

{
  "test-index" : {
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "test-ilm-policy"
        },
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "test-index",
        "creation_date" : "1638365760565",
        "priority" : "100",
        "number_of_replicas" : "1",
        "uuid" : "4LdlNUg-Q9WB81heA4b94Q",
        "version" : {
          "created" : "7150299"
        }
      }
    }
  }



ILM settings : 


{
  "test-ilm-policy" : {
    "version" : 3,
    "modified_date" : "2021-12-01T13:32:49.370Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "300kb",
              "max_primary_shard_size" : "10gb",
              "max_age" : "1d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "2d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    },
    "in_use_by" : {
      "indices" : [
        "test-index",
      ],
      "data_streams" : [ ],
      "composable_templates" : [ ]
    }
  }
}

Template: 


{
  "my_template" : {
    "order" : 0,
    "index_patterns" : [
      “test-index-*”
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : “test-ilm-policy",
          "rollover_alias" : "test-index”
        },
        "number_of_shards" : "1",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}
4

@warkolm I have shared the details that you asked.

5

I suspect perhaps You need to create the initial managed index that maps the writer alias to a concrete index.

See Here

Yours would look something like

PUT test-index-2021.12.01
{
  "aliases": {
    "test-index":{
      "is_write_index": true 
    }
  }
}

Also

            "rollover" : {
              "max_size" : "300kb", <!----- This is not going to work well
              "max_primary_shard_size" : "10gb",
              "max_age" : "1d"

ILM is mean to work on the scale of GBs etc. so it will not rollover exactly on 300KB etc. I have written a bit about that here

Also you will never want 300KB indices that is very small and inefficient.

Well typically I would suggest starting with the defaults from say filebeat where all this is already configured... and works out of the box... get used to how elastic works.

Today we suggest Shard Sized based Rollover ... Time Based / Daily has some usefulness but can end up with many small indices and shards which can be wasteful.

And of course there are some nice docs here and here

6

@stephenb Thank you for your answer. 300kb, I set for testing rollover, for faster rollover. please ignore that.

I got your point, what I'm trying to do is set roll over via filebeat(using date) and also set additional roll over via shrads or size. How can I achieve that?

7

It does not really work that way... either you are using ILM or date based rollover in filebeat with index names not both.

The closest is to use ILM set Max Age 1d and a Max Shard Size.. that way it will either roll over 1 a day or or busier indices rollover at the shard size.

8

Thank you @stephenb

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.