Add ILM to existing index 7.17.9

Elastic Stack Elasticsearch
1

I have an existing index I've applied the 30 day default lifecycle management policy to. But it doesn't seem to be working. I've read through the documentation and I/m obviously missing something.

The index I'm attempting to add the lifecycle to was not created with a template. Could that be my problem? If so, is there a way to build a template from the index - then use this template to launch a new index and apply the lifecycle management policy to that?

I don't see a way to create a template from an index.
I don't see a way to apply a template (for ilm purposes) to an existing index.

This is how the index was created:

PUT /dfbi.log-prod
{
  "settings": {
    "index": {
      "routing": {
        "allocation": {
          "include": {
            "_tier_preference": "data_content"
          }
        }
      },
      "number_of_shards": "3",
      "number_of_replicas": "1"
      }
  }
}

the alias of 'dfbi.log-prod' was set up later

_ilm\explain shows me this:

{
  "indices" : {
    "dfbi.log-prod-000001" : {
      "index" : "dfbi.log-prod-000001",
      "managed" : true,
      "policy" : "30-days-default",
      "lifecycle_date_millis" : 1683152809675,
      "age" : "5.92d",
      "phase" : "hot",
      "phase_time_millis" : 1683216402487,
      "action" : "rollover",
      "action_time_millis" : 1683216402687,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1683216402687,
      "phase_execution" : {
        "policy" : "30-days-default",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_primary_shard_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1681853018351
      }
    }
  }
}

Any help here is very much appreciated.

2

Did you see Configure a lifecycle policy | Elasticsearch Guide [8.7] | Elastic? That should let you attach the existing indices.

3

I'm running 7.17.9 - Will I be able to attach to an existing index with this version?

Forgive me, I'm the Linux Admin here - I haven't used ELK much - I set up the hardware, got the cluster running, and am doing my best to answer user questions and set up parameters required by our ELK users.

In this 3 node cluster, we're just running one index. All data from our production platform is loading into this one index. After I set up the lifecycle management policy, I expected that after the hot period the lifecycle management processes would strip out the 'old' entries in the index and put them into a new, warm, version of the index that queries could be run against if my colleagues need access to older logs. Am I correct in this? What should I be seeing in Kibana (or elsewhere) to let me know lifecycle management is working?

4

Yep, it's version agnostic.

Your theory is sound! We'd need to see the actual ILM policy to comment if your implementation is correct.

5

ILM assumes that you are using time based indices ,e.g. through rollover or a data stream, and manages the lifecycle by moving and deleting complete indices. It does not delete partial data from within an index or move data between indices.

6

Ahh yeah good catch!

7

Thanks guys - that was my suspicion. We have just one big index that keeps growing. On our old (now test) instance running an old version I have cron job that just deletes the index once a month.
I will see what it takes to get this set up properly. I can google myself, but it could be helpful if you point me to a resource that goes through setting up the indexes/templates/policies from scratch in a way that will support ilm? Much appreciated!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.