ILM rollout not working

Elastic Stack Elasticsearch
1

Hi all,

We want to active the rollout into hot phase in a index. Policy applied before rollout:

PUT _ilm/policy/test-custom
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 10
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          }
        }
      },
      "delete": {
        "min_age": "5d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

And work fine: 1 day into hot phase, 4 into warm phase and delete at 5 day. Then try to apply this policy, adding rollout into hot phase:

PUT _ilm/policy/test-custom
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "5gb",
            "max_age": "1d"
          },
          "set_priority": {
            "priority": 10
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          }
        }
      },
      "delete": {
        "min_age": "5d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

The problem is: index stay into hot phase and never enter into warm phase or delete it.

We try to not specify max_age and the same result.

Some idea? Thanks

2

Hi,

Please to check if the ILM policy is applied and check the output of this command,

GET index/_ilm/explain

and as per the doc

Create a lifecycle policy
edit

A lifecycle policy specifies the phases in the index lifecycle and the actions to perform in each phase. A lifecycle can have up to five phases: hot, warm, cold, frozen, and delete.

For example, you might define a timeseries_policy that has two phases:

    A hot phase that defines a rollover action to specify that an index rolls over when it reaches either a max_primary_shard_size of 50 gigabytes or a max_age of 30 days.
    A delete phase that sets min_age to remove the index 90 days after rollover.

The min_age value is relative to the rollover time, not the index creation time.
3

Hi, seem like there're some problem to do rollout:

{
  "indices" : {
    "filebeat-7.16.2-service-2022.05.24" : {
      "index" : "filebeat-7.16.2-service-2022.05.24",
      "managed" : true,
      "policy" : "custom",
      "lifecycle_date_millis" : 1653350421595,
      "age" : "8.14h",
      "phase" : "hot",
      "phase_time_millis" : 1653350720388,
      "action" : "rollover",
      "action_time_millis" : 1653350955063,
      "step" : "ERROR",
      "step_time_millis" : 1653353138878,
      "failed_step" : "check-rollover-ready",
      "is_auto_retryable_error" : true,
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "index.lifecycle.rollover_alias [filebeat-7.15.2] does not point to index [filebeat-7.16.2-service-2022.05.24]",
        "stack_trace" : """java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [filebeat-7.15.2] does not point to index [filebeat-7.16.2-service-2022.05.24]
	at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:156)
	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:226)
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:408)
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:339)
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:186)
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:220)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
"""
      },
      "phase_execution" : {
        "policy" : "custom",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "5gb",
              "max_age" : "1d"
            },
            "set_priority" : {
              "priority" : 10
            }
          }
        },
        "version" : 46,
        "modified_date_in_millis" : 1653292239802
      }
    }
  }
}

We deploy the index by the index template filebeat-7.15.2:

{
  "index": {
    "lifecycle": {
      "name": "custom",
      "rollover_alias": "filebeat-7.15.2"
    },
    "mapping": {
      "total_fields": {
        "limit": "10000"
      }
    },

We create the index in the logstash ingest:

    output {
      elasticsearch {
        user => writter_user
        password => "${LOGSTASH_BEAT_PASSWORD}"
        hosts => ["https://elasticsearch-es-http:9200"]
        index => "filebeat-%{[@metadata][version]}-%{[@metadata][target_index]}-%{+YYYY.MM.dd}"
        cacert => 'ca.crt'
      }
    }

This configuration create a index by service every day.

Some idea of how to apply this rollout configuration?

Thaks and regards,

4

Hi @gerardgorrion Thank you for posting the details,

as per the error below, the issue seems pretty clear

the issue is related to rollover_alias name, they are not matched with index name you have and the policy will never be applied. apparently this happened after the upgrade of the filebeat.

i suggest to modify the rollover_alias to filebeat if you're using only one index template for filebeats shippers .

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.