Lifecycle policies don't work

Elastic Stack Elasticsearch
1

As per the title, ILM simply does not work, or at least it doesn't work according to what Kibana or the docs make you believe should work.

I can create a lifecycle policy and attach a index but it will never roll over. I also tried doing it as per the docs. Create a template, create the policy, attach template to policy, manually create the first index and then have logstash index to myindex-000001 along with the ilm policy settings in the output but that too creates no rollover indexes.

The legibility of the docs is not good. Required settings are hard to spot and too strung out over long pages.

2

It'd be helpful if you could share your policy, your template and an explain of the policy that's not working.

3 
PUT _index_template/v2-testdiag
{
  "version": 1,
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "test-diag",
          "rollover_alias": "v2-testdiag"
        },
        "number_of_shards": "1",
        "number_of_replicas": "0"
      }
    },
    "mappings": {
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "_routing": {
        "required": false
      },
      "dynamic": false,
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "name": {
          "type": "keyword"
        },
        "message": {
          "type": "text"
        },
        "type": {
          "type": "keyword"
        }
      }
    },
    "aliases": {
      "v2-testdiag": {}
    }
  },
  "index_patterns": [
    "v2-testdiag-*"
  ]
}

{
  "test-diag" : {
    "version" : 7,
    "modified_date" : "2021-06-17T04:24:05.152Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "20gb",
              "max_age" : "1m"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "5m",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    }
  }
}

    elasticsearch {
      hosts => ["localhost"]
      index => "v2-testdiag-000001"
      ilm_rollover_alias => "v2-testdiag"
      ilm_pattern => "000001"
      ilm_policy => "test-diag"
    }
4

Can you share an explain of the policy?

5

I want to create a new index every minute and delete indexes that are older than 5 minutes.

6

Sorry, I meant Explain lifecycle API | Elasticsearch Guide [7.13] | Elastic :slight_smile:

7

Like this?

{
  "indices" : {
    "v2-testdiag-000002" : {
      "index" : "v2-testdiag-000002",
      "managed" : true,
      "policy" : "test-diag",
      "lifecycle_date_millis" : 1623906761941,
      "age" : "18.86m",
      "phase" : "hot",
      "phase_time_millis" : 1623906762121,
      "action" : "rollover",
      "action_time_millis" : 1623907294744,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1623907294744,
      "phase_execution" : {
        "policy" : "test-diag",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "20gb",
              "max_age" : "1m"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 7,
        "modified_date_in_millis" : 1623903845152
      }
    },
    "v2-testdiag-000001" : {
      "index" : "v2-testdiag-000001",
      "managed" : true,
      "policy" : "test-diag",
      "lifecycle_date_millis" : 1623906761766,
      "age" : "18.87m",
      "phase" : "delete",
      "phase_time_millis" : 1623907294560,
      "action" : "delete",
      "action_time_millis" : 1623907294560,
      "step" : "wait-for-shard-history-leases",
      "step_time_millis" : 1623907294560,
      "phase_execution" : {
        "policy" : "test-diag",
        "phase_definition" : {
          "min_age" : "5m",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        },
        "version" : 7,
        "modified_date_in_millis" : 1623903845152
      }
    }
  }
}

The reason there is now a second index is because I forced the lifecycle policy to run again. That created the second index but after that no new index is created nor is the old index deleted.

8

ILM is not really intended to be used on 1m lifecycle. I Do not believe You will achieve your desired behavior. My understanding is that ILM is an opportunistic background task it is not preemptive so it is not going to execute on the exact time frame.

It's designed to work on the order of hours or days not minutes.

9

If that is the case the UI shouldn't allow such values (and lower!) or at the very least give a warning it might not work as expected.

In my case this is just for testing so I don't have to wait days to see if the settings actually work.

10

Apologies. Ah I suspected This was for testing I've written several other threads on this topic where folks are trying to test using minutes and perhaps you're right The documents could be better.

When you use normal numbers the ILM will work plus or minus a couple percents in either time or size of the index.

You can always open an issue for Kibana and / or the documentation and supply suggestions that's part of the community and an open source project.

11

I have given suggestions numerous times. I don't mind because in the end Elastic becomes a better product but at the same time I am a paying customer, I expect documentation to at least give me basic working examples. In case of ILM that simply isn't the case. I now have one policy that for some reason works and another one that doesn't even though I followed the exact same steps. Logstash trows an alias related error but than the next problem is the logstash documentation doesn't really explain how, or if you need to set up the ilm inside logstash as well.

At this point I pretty much wasted a day on trying to figure something out which should be very simple. Your UI lets you create a policy and link a index. It doesn't mention anything else being required but that simply won't work.

12

That's understandable. Did you raise this with the Support team?

13

@Sjaak01 ILM uses a poll interval of 10 minutes (indices.lifecycle.poll_interval) which defaults to 10 minutes.

The times in the phases are expected to be (considerably) larger than that, as @stephenb said. Your rollover with 1m and deletion with 5m is not helping that and could cause issues.

We don't recommend changing this value for production, but for a test you could set it to 1 minute.

Make your rollover be 2 or 3 minutes and your deletion 5 minutes.

Also your Logstash output is pointing to the ILM settings AND to the index name. So there might be a conflict, or simply the ILM settings are being ignored, making ingestion always happen in v2-testdiag-000001, not in the alias, which should change after rollover.

Remove the index name in the test. Logstash will create the index with the proper name and policy and actually write to the rollover alias, not to the first index as you specified.

1 Like
14

I want to apoligize because I was overly rude in my previous post.

I gave it some time and looked at it again and everything is working now. As mentioned before, the ILM can be a bit slow so for anyone that wants to test, I suggest 1 hour intervals.

What I had to do is is 1) create a index template 2) create the ILM 3) manually create the first index matching the template. The latter seems to be important for ILM to actually work.

It might be a idea to make the explanation in the GUI a bit clearer because its easy to think just setting it up using the UI is enough.

1 Like
15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.