My lifecycle policy is not working

[its-ogawa]

I want to automatically delete indexes that are 60 days old and have set up the following lifecycle policy.

# curl -XGET '_ilm/policy/its_index-policy?pretty'
{
  "its_index-policy" : {
    "version" : 1,
    "modified_date" : "2023-02-21T08:37:18.466Z",
    "policy" : {
      "phases" : {
        "warm" : {
          "min_age" : "5d",
          "actions" : { }
        },
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "20gb",
              "max_age" : "2d",
              "max_docs" : 50000000
            }
          }
        },
        "delete" : {
          "min_age" : "60d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    },
    "in_use_by" : {
      "indices" : [
      
      ... snip ...
      
      ],
      "data_streams" : [ ],
      "composable_templates" : [
        "its_index_template"
      ]
    }
  }
}

However, there are still indexes that are 60 days old.

# curl -XGET 'example-2023.02.22/_ilm/explain?pretty'
{
  "indices" : {
    "example-2023.02.22" : {
      "index" : "example-2023.02.22",
      "managed" : true,
      "policy" : "its_index-policy",
      "index_creation_date_millis" : 1677054544127,
      "time_since_index_creation" : "75.75d",
      "lifecycle_date_millis" : 1677054544127,
      "age" : "75.75d",
      "phase" : "hot",
      "phase_time_millis" : 1683599336458,
      "action" : "rollover",
      "action_time_millis" : 1677054544636,
      "step" : "ERROR",
      "step_time_millis" : 1683599936449,
      "failed_step" : "check-rollover-ready",
      "is_auto_retryable_error" : true,
      "failed_step_retry_count" : 5454,
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "index.lifecycle.rollover_alias [its_index-policy-alias] does not point to index [example-2023.02.22]"
      },
      "phase_execution" : {
        "policy" : "its_index-policy",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "20gb",
              "max_age" : "2d",
              "max_docs" : 50000000
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1676968638466
      }
    }
  }
}

What am I doing wrong?

[stephenb (Stephen Brown)]

What version are you on?

What are all the steps you followed to get to this point?

The error states

        "reason" : "index.lifecycle.rollover_alias [its_index-policy-alias] does not point to index [example-2023.02.22]"

Your writes alias is not correct so the index can not rollover... no rollover not moving on to the next phase.

What does this show

GET its_index-policy-alias

[its-ogawa]

Thank you for your prompt response.

The version of elasticsearch is 8.6.2.

{
  "name" : "ITS-ELST-01",
  "cluster_name" : "els-cluster",
  "cluster_uuid" : "VhwbgOduQlqwvUB_1t1HiA",
  "version" : {
    "number" : "8.6.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "2d58d0f136141f03239816a4e360a8d17b6d8f29",
    "build_date" : "2023-02-13T09:35:20.314882762Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.2",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

The circumstances that led me to this problem were that I encountered the following logs.

# tail /var/log/messages

... snip ...

May  9 12:23:46 ITS-ELST-01 logstash[1597324]: [2023-05-09T12:23:46,424][WARN ][logstash.outputs.elasticsearch][main][d990eee626c9a5b7ee766183d9ecad6781d64a3cc7bd99486634821aa1cb3bb4] Could not index event to Elasticsearch. status: 400, ... snip ...l, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [1] shards, but this cluster currently has [4000]/[4000] maximum normal shards open;"}}}

I thought I was in an environment with no more than 4000 shards, but it turns out that the index life cycle is apparently not working.

Also, the result of its_index-policy-alias is as follows.

# curl -XGET '/its_index-policy-alias?pretty'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [its_index-policy-alias]",
        "resource.type" : "index_or_alias",
        "resource.id" : "its_index-policy-alias",
        "index_uuid" : "_na_",
        "index" : "its_index-policy-alias"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [its_index-policy-alias]",
    "resource.type" : "index_or_alias",
    "resource.id" : "its_index-policy-alias",
    "index_uuid" : "_na_",
    "index" : "its_index-policy-alias"
  },
  "status" : 404
}

[stephenb (Stephen Brown)]

So you have a couple problems..

ILM works with a rollover alias

You have no valid rollover alias

Clients like logstash the elasticsearch output should be writing to the the rollover alias not the concret index

Perhaps take a look at

Rollover | Elasticsearch Guide [8.7] | Elastic

Also share your logstash conf particularly the output section..

In short you are not writing to a rollover alias therefore the index can not rollover therefore ILM will not work.

[its-ogawa]

The logstash conf is as follows:

# The # character at the beginning of a line indicates a comment.
# Use comments to describe your configuration.
input {
    beats {
        port => "5044"
    }
}
# The filter part of this file is commented out to indicate that
# it is optional.
filter {
... snip ...
}
output {
    if([fields][index_name]) {
      elasticsearch {
        hosts => ["localhost"]
        index => "%{[fields][index_name]}-%{+YYYY.MM.dd}"
      }
    }
    else {
      elasticsearch {
        hosts => ["localhost"]
        index => "other-%{+YYYY.MM.dd}"
      }
    }
}

[stephenb (Stephen Brown)]

You need to read the conceptd in the doc

Tutorial: Automate rollover with ILM | Elasticsearch Guide [8.7] | Elastic

You need to setup an initial managed index with the write rollover alias

Tutorial: Automate rollover with ILM | Elasticsearch Guide [8.7] | Elastic

# Create Initial Managed index 
PUT my-index-2023.05.06-000001
{
  "aliases": {
    "its_index-policy-alias":{
      "is_write_index": true 
    }
  }
}

Then you need to point logstash to write to the write alias.

This only does daily index not ILM with rollover which will not work

 index => "other-%{+YYYY.MM.dd}"

Should be writing to the alias

index => "its_index-policy-alias"

[its-ogawa]

Thank you for the information you have provided.
I am still trying to understand the rollover alias.
I maintain a number of logs and have defined an index_name in filebeat for each.
If I want to use that index_name, how do I do it?

In the example above, I would write

index => "%{[fields][index_name]}-%{+YYYYY.MM.dd}"

[stephenb (Stephen Brown)]

When you use rollover alias is does not include the date...

Read the docs I actually gave you the solution...

Perhaps look at this

Filebeat process different log paths and write data to seperate index,Without use of logstash and follow ILM/rollover alias defined in template

@PRASHANT_MEHTA Looks at this carefully this is minimal but does everything you ask. This is the order you need to do this as well ... BTW timeseries indices are of the pattern my-index-yyyy.mm.dd-000001 # Create Common ILM Policy PUT _ilm/policy/mis-monitoring-common-policy { "policy": { "phases": { "hot": { "actions": { "rollover": { "max_age": "1d", "max_size": "50gb" }, "set_priority": { "priority": 10…

[its-ogawa]

@stephenb

When you use rollover alias is does not include the date...

That surprises me!

Even without the {+YYYY.MM.dd}, is it still managed by date in the index?

---

I actually tried to do this.

I can't see the index date...

# curl -XGET http://localhost:9200/_cat/indices?v
health index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open example ... snip ...

Does this mean that it will keep logging for one index for a given lifecycle?

---

How do I create rollover_alias?

@stephenb

# Create Initial Managed index 
PUT my-index-2023.05.06-000001
{
  "aliases": {
    "its_index-policy-alias":{
      "is_write_index": true 
    }
  }
}

Also, I already manage nearly 4000 indexes.
Do I need to write a statement to apply rollover_alias to all indexes?

[its-ogawa]

I created several new policies and worked on understanding ILM.
Stephen eagerly tried to show me how to solve it, but I finally couldn't master it.

Instead, I enlisted the help of Curator. This worked out well, and I am very happy with the results.

delete_indices | Curator Reference [8.0] | Elastic

I hope this helps others who are having the same problem.

[stephenb (Stephen Brown)]

@its-ogawa

Also, I already manage nearly 4000 indexes.

You need to clean up indices to make room

@its-ogawa

Do I need to write a statement to apply rollover_alias to all indexes?

Apologies, but no all those indices are too late for rollover I was trying to give you the solution for "Go Forward"

If you wanted to manage existing then you could have looked at this

Manually Create a Policy

Manage existing indices | Elasticsearch Guide [8.11] | Elastic

And Manually Applied to every Index

Index lifecycle management settings in Elasticsearch | Elasticsearch Guide...

Which then would parse the date of the index and apply the ILM Policy ... that would not work for new indices only old..

This may still work for you....

@its-ogawa

Instead, I enlisted the help of Curator. This worked out well, and I am very happy with the results.

Good!

I think you could have done the same with the manual ILM but glad you got it working...