Does single node Elasticsearch supports ILM ploicy

Ravi_Pattar · December 29, 2023, 11:46am

Hello,

I have applied the ILM policy because the disk space usage was seen very high.

But I don't see any changes in the disk space after implementing the ILM policy for filebeat.

shards disk.indices disk.used disk.total disk.percent
   100                                               
   111      199.1gb   205.4gb      242gb           84

207G    /
202G    /var
201G    /var/lib
200G    /var/lib/elasticsearch/nodes/0/indices
200G    /var/lib/elasticsearch/nodes/0
200G    /var/lib/elasticsearch/nodes
200G    /var/lib/elasticsearch
15G     /var/lib/elasticsearch/nodes/0/indices/dE43jALIRbmNM5jBMj8ZDw/0/index
15G     /var/lib/elasticsearch/nodes/0/indices/dE43jALIRbmNM5jBMj8ZDw/0
15G     /var/lib/elasticsearch/nodes/0/indices/dE43jALIRbmNM5jBMj8ZDw

Thanks,

leandrojmp · December 29, 2023, 1:47pm

What does your policy looks like?

You need a delete phase so your data would be deleted after some time to free up space.

Ravi_Pattar · December 29, 2023, 2:18pm

Hot phase (Required)
Rollover: enabled
Maximum primary shard size: 50G
Maximum age: 7 days

Warm phase
Move data into phase when:5 days old

Cold phase
Move data into phase when:10 days old

Delete phase
Move data into phase when:25 days old

leandrojmp · December 29, 2023, 2:22pm

You have just one node, you do not need a warm and cold phase as this would do nothing, you can remove those steps.

Also, assuming that your data will rollover after 7 days, it may take 32 days for it to be deleted, you need to check if this time range is enough.

From what you shared it seems that you have a lot of data for a small cluste and may need to reduce both the time that it takes to delete and rollover, and the shard size.

Ravi_Pattar · December 29, 2023, 2:50pm

Hello Leandro,

Thanks for the response. Please confirm if I can go with the below new changes as this is a production server.

Hot phase (Required)
Rollover: enabled
Maximum primary shard size: 10G
Maximum index size: 10G
Maximum age: 7 days

Delete phase
Move data into phase when:15 days old

Also do we need to reboot the server after the changes?

stephenb · December 29, 2023, 3:34pm

Yes, that seems reasonable. But keep in mind that the following phase and your case the delete phase is calculated from the time of rollover which you have set as a maximum of 7 days in the hot.

So index take 7 days to roll over plus 15 days until delete. You could have data that's 22 days old and then it will delete the 7 days worth of data since the index contain 7 days where the data.

If you're indices, fill up faster. Rollover faster. The same math implies just with a shorter duration

But yes, that's all seems reasonable

No, you do not need to reboot the system.

You may need to force a rollover on the existing indices depending on what version of Elasticsearch and the ILM phase they're in, in order to pick up/ apply the new ILM policy.

Ravi_Pattar · December 29, 2023, 3:52pm

Thanks, Stephen.

What I understand from the above is I can have the data available for last 22 days and once the delete operation occurs I hope the latest 7 days data would be available.

Also the ELK version here is 7.17.15

stephenb · December 29, 2023, 4:09pm

Let say your indices look like this, rolling over every 7 days

my-index-000001 Jan 1 - Jan 7
my-index-000002 Jan 8 - Jan 15
my-index-000003 Jan 16 - Jan 23
my-index-000004 Jan 24 - Feb 1
....

On Jan ~22nd (15days after rollover on Jan 7) my-index-000001 with data Jan 1 - Jan 7th would be deleted

The other indices would still be there.

The best way to do this is to get started, observe, and adjust.

Ravi_Pattar · December 29, 2023, 5:57pm

Thank you Stephen for giving more clarity.

Ravi_Pattar · January 4, 2024, 7:49am

Hello,

As discussed in the previous post that ILM policy for filebeat was created on 29th Dec-23 as below.

Hot phase (Required)
Rollover: enabled
Maximum primary shard size: 10G
Maximum index size: 10G
Maximum age: 7 days

Now, I am seeing the below message on the kibana GUI which is a production server.

You are editing an existing policy. Any changes you make will affect 32 linked indices(opens in a new tab or window) and **1 linked index template** that are attached to this policy. Alternatively, you can save these changes in a new policy. And 1 linked index template it shows as filebeat-7.17.15.

Question: Is anything wrong here? Please suggest. I am worried if the 1 linked template as mentioned also gets deleted and create a system wide issue.

Thanks,

stephenb · January 4, 2024, 5:32pm

Ravi_Pattar:

Now, I am seeing the below message on the kibana GUI which is a production server.
You are editing an existing policy. Any changes you make will affect 32 linked indices(opens in a new tab or window) and **1 linked index template** that are attached to this policy. Alternatively, you can save these changes in a new policy. And 1 linked index template it shows as filebeat-7.17.15. 
Question: Is anything wrong here? Please suggest. I am worried if the 1 linked template as mentioned also gets deleted and create a system wide issue.

If you are seeing this?

No that is just a standard awareness warning.
The policy change can/will affect the already linked indices.
The template will not be deleted or changed in any way.
New Indices created by the template will have this new policy applied.

I do want to be clear this is your production system ... if you are concerned you should test in a non-production system.

Ravi_Pattar · January 5, 2024, 5:23am

Thanks, Stephen

Ravi_Pattar · January 5, 2024, 5:28am

Yeah, this is on the production server. Also I have implemented the same ILM policy for filbeat in test instance before applying the ILM in production server.

Below is the ss from my test instance where it is not showing anything for 1 linked index template

# Edit policy filebeat

**You are editing an existing policy.** Any changes you make will affect [4 linked indices that are attached to this policy. Alternatively, you can save these changes in a new policy.

Save as new policy

Ravi_Pattar · January 6, 2024, 6:16pm

Hi @stephenb

Per our earlier discussions on this thread. The ILM policy was set for filebeat index as below on 29th Dec-23.

    Hot phase (Required)
    Rollover: enabled
    Maximum primary shard size: 10G
    Maximum index size: 10G
    Maximum age: 7 days

    Delete phase
    Move data into phase when:15 days old

Note: I was expecting the last 7 days data should have been deleted by now but that's not the case because I could see no changes.

Below details on the ILM set.

Correct me if I am wrong on my understanding here. Is it going to take 22 days to run the delete operation? when the max age was set to 7 days in hot phase and 15 days to move data into delete phase.

If this is a case then can I change it to lesser value other than 15 days e.g 5 or < 10 days etc in the delete phase instead of waiting for another 15 days more because I can see that the disk space utilization has started piling up.

GET _cat/allocation?v&s=disk.indices&h=shards,disk.indices,disk.used,disk.available,disk.total,disk.percent

shards disk.indices disk.used disk.total disk.percent
123
137 179.4gb 185.9gb 242gb 76

Also from the command output, du -h / | sort -rh | head -10

181G /var/lib/elasticsearch/nodes/0/indices
181G /var/lib/elasticsearch/nodes/0
181G /var/lib/elasticsearch/nodes
181G /var/lib/elasticsearch

Please suggest as this on production.

Thanks,

stephenb · January 6, 2024, 6:33pm

Hi @Ravi_Pattar Again this is your production system. I would highly recommend testing this on a non-production system.

Yes, of course, you can set delete to what you like...

Perhaps a suggestion

Rollover on 1 day...

Then Adjust the Delete phase to what you want... in Days... this will give you a lot of control.

After you set rollover to 1 fay, you will need to force a rollover on the current index so it does not wait to take effect.

Shortening the Delete phase should work after it is set...

Then you can optimize later (like Rollover on 3 Days) if needed.

Ravi_Pattar · January 9, 2024, 7:43am

Hi @stephenb

I did few changes on my test environment. e.g. below

On the Kibana GUI
Stack Management > Index Lifecycle policies and the filtered out for filebeat

Applied the ILM for filebeat as below

Hot phase (Required)
    Rollover: enabled
    Maximum primary shard size: 5G
    Maximum index size: 5G
    Maximum age: 1 days

    Delete phase
    Move data into phase when:2 days old

Current status when checked using the curl

# curl -X GET "localhost:9200/_cat/indices/filebeat*?v"
health status index                              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   filebeat-7.17.13-2023.12.29        bgI3JWqJQv-CF3URgOboXA   1   1     360230            0     40.7mb         40.7mb
yellow open   filebeat-7.17.13-2023.12.28        aN5saI_kRJmrXBTWhZ7aUg   1   1     157875            0     17.7mb         17.7mb
yellow open   filebeat-7.17.13-2023.12.27        3Ff2VmfjT72ElvbbUC3VyA   1   1     267396            0     29.8mb         29.8mb
yellow open   filebeat-7.17.13-2024.01.02        uuFIjZdkTYqO6RNEx9B9gg   1   1       4984            0      2.7mb          2.7mb
yellow open   filebeat-7.17.13-2024.01.03        MgKear7JQA6hYs5VdPdYxg   1   1       7668            0      2.4mb          2.4mb
yellow open   filebeat-7.17.13-2024.01.01        Kj2FCRBUQca5n3mg-yed7Q   1   1       2671            0      1.2mb          1.2mb
yellow open   filebeat-7.17.13-2024.01.04        85EFuavPQamUfnLCCu-Ozw   1   1       4507            0      2.6mb          2.6mb
yellow open   filebeat-7.17.13-2024.01.05        JbsEFg0tSJCGsJddNgbYgg   1   1       2928            0      1.5mb          1.5mb
yellow open   filebeat-7.17.13-2024.01.01-000005 9UnVNkJAS1aaUXEdrt2zZA   1   1          0            0       227b           227b
yellow open   filebeat-7.17.13-2023.12.26        MOlB9WyrSOaDDCf-CYJoJQ   1   1    2108355            0      235mb          235mb
yellow open   filebeat-7.17.13-2024.01.09-000006 oTXERMP3RFWyUDqy3DMj9A   1   1          0            0       227b           227b
yellow open   filebeat-7.17.13-2024.01.09        rIQ_RaOeSIuycyO-Twxizw   1   1       2621            0      1.5mb          1.5mb

Please suggest if there's anything wrong here and why I don't see delete is not occurring.

Thanks,

Ravi_Pattar · January 9, 2024, 1:11pm

Hello,

Can someone please help on my recent post?

Thanks,

stephenb · January 9, 2024, 3:06pm

First Please be patient. There are many questions and your question is no more important than anyone else's. It's only been a couple hours. We will get to your questions if and when we can. There is no guarantee that your question will even get answered. It's only volunteers here.

Run this for your policy

Please run these show the command and the result

GET _ilm/policy/<policy_id>

Then run

GET filebeat-7.17.13-2023.12.29/_ilm/explain

Ravi_Pattar · January 9, 2024, 6:08pm

stephenb:

GET _ilm/policy/<policy_id>

Hi Stephen,

Thanks, please find the details below.

GET _ilm/policy/filebeat

{
  "filebeat" : {
    "version" : 13,
    "modified_date" : "2024-01-09T12:23:33.729Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50mb",
              "max_primary_shard_size" : "49mb",
              "max_age" : "1d"
            }
          }
        },
        "delete" : {
          "min_age" : "1d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    },
    "in_use_by" : {
      "indices" : [
        "filebeat-7.17.13-2023.12.29",
        "filebeat-7.17.13-2023.12.28",
        "filebeat-7.17.13-2023.12.27",
        "filebeat-7.17.13-2023.12.26"
      ],
      "data_streams" : [ ],
      "composable_templates" : [ ]
    }
  }
}

Then run

GET filebeat-7.17.13-2023.12.29/_ilm/explain

{
  "indices" : {
    "filebeat-7.17.13-2023.12.29" : {
      "index" : "filebeat-7.17.13-2023.12.29",
      "managed" : true,
      "policy" : "filebeat",
      "lifecycle_date_millis" : 1703829688349,
      "age" : "11.49d",
      "phase" : "hot",
      "phase_time_millis" : 1704823122627,
      "action" : "rollover",
      "action_time_millis" : 1703829689068,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1704823122627,
      "is_auto_retryable_error" : true,
      "failed_step_retry_count" : 94,
      "phase_execution" : {
        "policy" : "filebeat",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50mb",
              "max_primary_shard_size" : "49mb",
              "max_age" : "1d"
            }
          }
        },
        "version" : 13,
        "modified_date_in_millis" : 1704803013729
      }
    }
  }
}

Thanks,

stephenb · January 9, 2024, 6:34pm

This index is Failing rollover... perhaps you changed the index name ... to daily indices...

Are you ingesting directly from Filebeat to Elasticsearch?

Share your entire filebeat.yml please...