[HELP! ! ] About ILM (IndexLifecycleManagement) of ElasticSearch

[YUUTA.INOUE-JPN (YUUTA.INOUE)]

Hello from Japan
I have a question for you dear engineers
I'm using Elasticsearch 7.6.2 and want to remove the accumulated indexes
The created ILM policy rolls over at 50GB/30 days, and I created an ILM policy that deletes indexes 33 days after the rollover.
I can check the log that reflects this ILM policy in elasticSearch, but I'm having trouble with the problem that the index is not deleted after 33 days.

<[2023-03-02][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [host] updating index lifecycle policy [winlogbeat]>

I created the same ILM policy on the validation server and the deletion worked fine
The diagram below shows the same policy applied (left diagram is NotWorking, right diagram is ILM working)

Both are rolled over indexes just before delete, but if you check the server where ILM is not running, the current phase is [completed] and the current status is also [completed]. I'm guessing this is why ILM doesn't work, but I'm not sure

please i am waiting for your help
Thank you

[system]

Elasticsearch 7.6 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

[stephenb (Stephen Brown)]

Hi @YUUTA.INOUE-JPN
Deletion phase is calculated from the rollover date not creation date.

So say rollover over was 30 days.. then delete 33 days ...

So the data would be deleted 33 days after it rolled over, not from the creation date. So it could be up to 63 days.

And your cluster is incredibly out of date. You should update as a matter of urgency.

[YUUTA.INOUE-JPN (YUUTA.INOUE)]

hello @stephenb

Thank you for your reply.
It is set to rollover at 30 days or 50GB, but in reality it is faster to reach 50GB, so many 50GB indexes are generated.
What I want you to pay attention to is where Current Action and Current phase are 【Completed】.
The current action of the rolled over index of the server where ILM is working normally and the deletion is being executed is 【hot】and the current phase is 【complete】.
I think that the reason why the deletion phase of ILM does not work is affected by the reason around here, but I do not know the details.

best regards
thank you

[stephenb (Stephen Brown)]

It is very hard to debug with screen shots.. in fact I can not read them..

Please provide in text the ILM policy and the ILM explain.

Also apologies I can not read the Japanese Characters ...we do have some people that can.

@YUUTA.INOUE-JPN

but if you check the server where ILM is not running,

Also I don't know what this means...why is ILM not running or do you mean not working?

[YUUTA.INOUE-JPN (YUUTA.INOUE)]

Im really sorry for my lack of explanation.
I just want to delete the rolled over index

Here's what the words mean
ILM working : ILM with added delete phase works and indexes are deleted
Not working : Indicates that the ILM-adapted index (with the delete phase enabled) will not be deleted.

These two compare indexes that have been rollover by an ILM and are waiting to be deleted.
The difference between the two is that I applied the same policy, but I noticed that the Elasticsearch server where the deletion operation does not work is displayed as [Completed]

The two servers are in exactly the same environment and have the same ILM policy applied

It is the right figure of the server for verification. The deletion was performed successfully on the verification server

I applied the same ILM policy to another server, but the current phase and current action are shown as 【Completed】and the delete action is not executed.

What steps should be taken to apply an ILM policy with a delete function to an index containing an index in the 【Completed】phase？

Please feel free to let me know what additional information you need
If my English is difficult to understand, please feel free to get angry.
thank you

yuuta

[stephenb (Stephen Brown)]

Your English is Excellent! My Japanese ... non-existent!

In the system that is not working.

Can you provide the actual ILM and ILM explain for the specific index on t

GET _ilm/policy/<yourpolicy>
GET <yourindex>/_ilm/explain

example 

GET _ilm/policy/filebeat
GET .ds-filebeat-8.4.3-2023.03.01-000422/_ilm/explain

[YUUTA.INOUE-JPN (YUUTA.INOUE)]

hello @stephenb

I would appreciate it if you could check the information
↓This is the ILM policy that doesn't perform the deletion that I'm having trouble with

GET _ilm/policy/<yourpolicy>


{
  "winlogbeat" : {
    "version" : 12,
    "modified_date" : "2023-03-01T23:54:54.756Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "delete" : {
          "min_age" : "33d",
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  }
}



GET winlogbeat-7.6.2-2023.01.06-000002/_ilm/explain


{
  "indices" : {
    "winlogbeat-7.6.2-2023.01.06-000002" : {
      "index" : "winlogbeat-7.6.2-2023.01.06-000002",
      "managed" : true,
      "policy" : "winlogbeat",
      "lifecycle_date_millis" : 1674525721345,
      "age" : "37.96d",
      "phase" : "completed",
      "phase_time_millis" : 1674525723677,
      "action" : "completed",
      "action_time_millis" : 1674525723677,
      "step" : "completed",
      "step_time_millis" : 1674525723677,
      "phase_execution" : {
        "policy" : "winlogbeat",
        "version" : 1,
        "modified_date_in_millis" : 1670833920987
      }
    }
  }
}

↓This is an ILM policy that is successfully deleted

GET _ilm/policy/winlogbeat

{
  "winlogbeat" : {
    "version" : 23,
    "modified_date" : "2023-03-03T01:30:53.737Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_age" : "1d"
            }
          }
        },
        "delete" : {
          "min_age" : "2d",
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  }
}




GET winlogbeat-7.6.2-2023.02.23-000051/_ilm/explain
{
  "indices" : {
    "winlogbeat-7.6.2-2023.02.23-000051" : {
      "index" : "winlogbeat-7.6.2-2023.02.23-000051",
      "managed" : true,
      "policy" : "winlogbeat",
      "lifecycle_date_millis" : 1677171648770,
      "age" : "7.35d",
      "phase" : "cold",
      "phase_time_millis" : 1677214908643,
      "action" : "complete",
      "action_time_millis" : 1677171649643,
      "step" : "complete",
      "step_time_millis" : 1677214908643,
      "phase_execution" : {
        "policy" : "winlogbeat",
        "phase_definition" : {
          "min_age" : "12h",
          "actions" : { }
        },
        "version" : 19,
        "modified_date_in_millis" : 1677645985045
      }
    }
  }
}

best regards
thank you

[stephenb (Stephen Brown)]

Hi @YUUTA.INOUE-JPN

So I am not exactly sure what is happening but here are a few thoughts...

First, even though you mentioned it in your first post I did no really think about it you are on 7.6 which is very very old.

There have been 18 significant releases and probably double that patch / bug fix releases.

Many Many improvements have been made on ILM in that time

What I think may be happening is that the Index is still under control of the First Version of the ILM Policy...

The index that is not working in the _ilm/explain

      "phase_execution" : {
        "policy" : "winlogbeat",
        "version" : 1, <!---- It still thinks it is under the first version of the policy
        "modified_date_in_millis" : 1670833920987
      }
    }

The policy

GET _ilm/policy/<yourpolicy>


{
  "winlogbeat" : {
    "version" : 12, <!------ Newest Version
    "modified_date" : "2023-03-01T23:54:54.756Z",

Policies can be updated and I know in newer versions of the Elastic Stack that works very well perhaps there is a bug in such an old version OR if the index already entered a certain state then the new policy may not take effect.

Also, the newer version of the _ilm/explain provide more information to debug with so that makes it hard to tell too...

Lifecycle policy updates | Elasticsearch Guide [7.6] | Elastic

I suspect if you created a new index with your new policy it would probably work as expected.

Hope this makes some sense

[YUUTA.INOUE-JPN (YUUTA.INOUE)]

@stephenb
Good evening from japan
Following your advice, I created a new index policy for the Completed index and replaced it.
The index was successfully deleted, saving disk space.

I did some research on "Completed". "complete" indicates that the ILM policy has been applied to the index, but the index migration process has not yet completed. In other words, the ILM policy has been applied and actions have been triggered, but not all actions have completed yet.

On the other hand, 'completed' indicates that the ILM policy has been applied to the index and all migration actions have completed. In other words, a "completed" index means that no more actions can be added to it.

Without knowing this, I implemented Elasticsearch's ILM.
I am reflecting on this.
I am still a 23-year-old young engineer and I feel sorry for my low technical skills.
Thank you for teaching me kindly

[stephenb (Stephen Brown)]

Hi @YUUTA.INOUE-JPN

@YUUTA.INOUE-JPN

I am still a 23-year-old young engineer and I feel sorry for my low technical skills.
Thank you for teaching me kindly

You are doing Excellent!! ... Good for you digging in on the tech ... We Learn by Doing ... keep it up!