Some problems about index life cycle

Elastic Stack Elasticsearch
1

I use logstash to add fields to dynamically generate indexes.

index => "%{[service]}-%{+YYYY.MM.dd}"

I configure the following index template to match the generated index

{
  "index_templates" : [
    {
      "name" : "pre-docker",
      "index_template" : {
        "index_patterns" : [
          "docker_console_pre*"
        ],
        "composed_of" : [
          "games-mappings",
          "pre-games-settings"
        ],
        "priority" : 112,
        "version" : 1
      }
    }
  ]
}

{
  "component_templates" : [
    {
      "name" : "pre-games-settings",
      "component_template" : {
        "template" : {
          "settings" : {
            "index" : {
              "lifecycle" : {
                "name" : "pre",
                "rollover_alias" : "docker_console_pre"
              },
              "routing" : {
                "allocation" : {
                  "require" : {
                    "box_type" : "hot"
                  }
                }
              },
              "number_of_shards" : "3",
              "number_of_replicas" : "1"
            }
          }
        },
        "version" : 1
      }
    }
  ]
}

When I was testing the rollover update in the index life cycle, I encountered several problems:

  1. Index template cannot write alias directly. It must append alias to index after index creation. Otherwise, the following error will be reported:

illegal_ argument_ exception: Rollover alias [docker_ console_ pre] can point to multiple indices, found duplicated alias [docker_ console_ pre] in index template [pre-docker]

  1. When the index starts to scroll, the 000001 index will be automatically deleted, but logstash will generate a new 00001 index and start writing data. It will report the error mentioned earlier soon.

  2. Since the index is generated once a day, the errors mentioned above also occur.

2

You will need to drop the %{+YYYY.MM.dd} from your output section, so that you are writing to the ILM alias name to start with.

What does your ILM policy look like?

3

If you delete %{+YYYY.MM.dd}, the roll alias will be the same as the index, and the life cycle strategy is as follows:

{
  "pre" : {
    "version" : 5,
    "modified_date" : "2021-05-03T15:20:13.108Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "25gb",
              "max_age" : "60s"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "120s",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        },
        "warm" : {
          "min_age" : "0ms",
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "box_type" : "warm"
              }
            },
            "set_priority" : {
              "priority" : 50
            }
          }
        }
      }
    }
  }
}
4

That's 60seconds, I don't think that is what you want. You may want to check the rest of your min_age settings as they all seem wrong.

No, because ILM defaults to adding numbers to the end of the provided alias. You need to be indexing data into the write alias that you defined.

5

The life cycle is short and is used for testing. Are the following steps correct:

  1. Configure lifecycle policy

  2. Configure index template

  3. Set the index of logstash to

index => "%{[service]}

  1. Delete the old index for logstash rebuild.

  2. See if the lifecycle works

6

If you've followed the documentation here - Configure a lifecycle policy | Elasticsearch Guide [7.12] | Elastic - then you should be on the right path. What you have outlined looks right from a high level.

Ok, but the reality is that it's not going to work with such short timeframes due to the way it's polled.

7

When I follow this procedure, logstash generates a docker_ console_ Index of pre. When scrolling updates, it prompts the following error:

llegal_ argument_ exception: index.lifecycle.rollover_ alias [docker_ console_ pre] does not point to index [docker_ console_ pre]

8

It'd be helpful if you could post a full reproduction of what you are doing.

9

Please refer to the life cycle policy and index template listed above. Logstash configuration is as follows:

output {
    elasticsearch {
        hosts => ["192.168.1.165:9200"]
        ssl => true
        ssl_certificate_verification => true
        cacert => "/etc/logstash/ssl/logstash.pem"
        index => "%{[service]}"
    }
}
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.