Policy for index failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]

I am using Logstash to create a daily index in ElasticSearch.

This time, I tried to create an index using an index template, but I noticed that ElasticSearch was generating the following ILM error.

$ tail -n 30 /var/log/elasticsearch/elasticsearch.log
... snip ...
[2021-07-08T15:09:12,202][ERROR][o.e.x.i.IndexLifecycleRunner] [ITS-ELST-01] policy [its_index-policy] for index [sro-msp-error-2021.07.08] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [its_index-policy-alias] does not point to index [sro-msp-error-2021.07.08]
        at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:124) [x-pack-core-7.12.0.jar:7.12.0]
        at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:175) [x-pack-ilm-7.12.0.jar:7.12.0]
        at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:335) [x-pack-ilm-7.12.0.jar:7.12.0]
        at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:273) [x-pack-ilm-7.12.0.jar:7.12.0]
        at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:184) [x-pack-core-7.12.0.jar:7.12.0]
        at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:217) [x-pack-core-7.12.0.jar:7.12.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
... snip ...

What does this error mean and how do I respond to it?


Just in case, I am attaching the description of the index policy and index template we are using.

$ curl -X GET http://localhost:9200/_ilm/policy/its_index-policy?pretty
{
  "its_index-policy" : {
    "version" : 1,
    "modified_date" : "2021-07-07T05:58:15.664Z",
    "policy" : {
      "phases" : {
        "warm" : {
          "min_age" : "5d",
          "actions" : { }
        },
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "20gb",
              "max_age" : "2d",
              "max_docs" : 50000000
            }
          }
        },
        "delete" : {
          "min_age" : "60d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    }
  }
}

$ curl -X GET http://localhost:9200/_index_template/its_index_template?pretty
{
  "index_templates" : [
    {
      "name" : "its_index_template",
      "index_template" : {
        "index_patterns" : [
          "sro-*",
          "max-*"
        ],
        "template" : {
          "settings" : {
            "index" : {
              "lifecycle" : {
                "name" : "its_index-policy",
                "rollover_alias" : "its_index-policy-alias"
              },
              "number_of_shards" : "1",
              "number_of_replicas" : "0"
            }
          },
          "mappings" : {
            "_source" : {
              "enabled" : true
            },
            "properties" : {
              "created_at" : {
                "format" : "EEE MMM dd HH:mm:ss Z yyyy",
                "type" : "date"
              },
              "host_name" : {
                "type" : "keyword"
              }
            }
          },
          "aliases" : {
            "its" : { }
          }
        },
        "composed_of" : [
          "logs-settings"
        ],
        "priority" : 500,
        "version" : 1,
        "_meta" : {
          "description" : "its index template"
        }
      }
    }
  ]
}

In addition, we will note the policy and configuration information for the logs in question.
Please check them if you need them.

$ curl -X GET http://localhost:9200/sro-msp-error-2021.07.08/_ilm/explain?pretty
{
  "indices" : {
    "sro-msp-error-2021.07.08" : {
      "index" : "sro-msp-error-2021.07.08",
      "managed" : true,
      "policy" : "its_index-policy",
      "lifecycle_date_millis" : 1625702406659,
      "age" : "6.55h",
      "phase" : "hot",
      "phase_time_millis" : 1625725154477,
      "action" : "rollover",
      "action_time_millis" : 1625702406825,
      "step" : "ERROR",
      "step_time_millis" : 1625725753591,
      "failed_step" : "check-rollover-ready",
      "is_auto_retryable_error" : true,
      "failed_step_retry_count" : 19,
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "index.lifecycle.rollover_alias [its_index-policy-alias] does not point to index [sro-msp-error-2021.07.08]",
        "stack_trace" : "java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [its_index-policy-alias] does not point to index [sro-msp-error-2021.07.08]\n\tat org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:124)\n\tat org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:175)\n\tat org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:335)\n\tat org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:273)\n\tat org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:184)\n\tat org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:217)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\n"
      },
      "phase_execution" : {
        "policy" : "its_index-policy",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "20gb",
              "max_age" : "2d",
              "max_docs" : 50000000
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1625637495664
      }
    }
  }
}

$ curl -X GET http://localhost:9200/sro-msp-error-2021.07.08/_settings?pretty
{
  "sro-msp-error-2021.07.08" : {
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "its_index-policy",
          "rollover_alias" : "its_index-policy-alias"
        },
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "sro-msp-error-2021.07.08",
        "query" : {
          "default_field" : [
            "message"
          ]
        },
        "creation_date" : "1625702406659",
        "number_of_replicas" : "0",
        "uuid" : "2oSbt7qWQMyY-TCSk5dN9g",
        "version" : {
          "created" : "7120099"
        }
      }
    }
  }
}

I'll add more.

Since I didn't know how to solve the problem, I decided to delete the lifecycle policy given to the index template.

Since I couldn't figure out how to delete only the lifecycle policy, I deleted the entire index template and created a new index template with only the minimum necessary information.

Just to be sure, I will write down the command.

$ curl -X DELETE "localhost:9200/_index_template/its_index_template?pretty"

$ curl -X PUT "localhost:9200/_index_template/its_index_template?pretty" -H 'Content-Type: application/json' -d'
{
  "index_patterns" : ["sro-*", "max-*"],
  "priority" : 1,
  "template": {
    "settings" : {
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  }
}
'

$ curl -X GET "localhost:9200/_index_template/its_index_template?pretty"

{
  "index_templates" : [
    {
      "name" : "its_index_template",
      "index_template" : {
        "index_patterns" : [
          "sro-*",
          "max-*"
        ],
        "template" : {
          "settings" : {
            "index" : {
              "number_of_shards" : "1",
              "number_of_replicas" : "0"
            }
          }
        },
        "composed_of" : [ ],
        "priority" : 1
      }
    }
  ]
}

What I care about is the lifecycle of the index, and I asked curator to play that role.

Is it correct to assume that lifecycle policy is a function to complete the index lifecycle in ElasticSearch, and curator is a replacement for it?