About the ILM policy implented and on the post observations

Elastic Stack Elasticsearch
1

Hello,

Recently I have implemented ILM policy on of the production setup. However, the rollover for the existing ILM policy seems to be working fine. But I am seeing indices for other version e.g. 8.x are utilizing the high disk space.

# date && du -h / | sort -rh | head -10
Fri Jan 26 05:42:50 UTC 2024
du: cannot access '/proc/193850/task/193850/fd/4': No such file or directory
du: cannot access '/proc/193850/task/193850/fdinfo/4': No such file or directory
du: cannot access '/proc/193850/fd/3': No such file or directory
du: cannot access '/proc/193850/fdinfo/3': No such file or directory
125G    /
119G    /var
118G    /var/lib
117G    /var/lib/elasticsearch/nodes/0/indices
117G    /var/lib/elasticsearch/nodes/0
117G    /var/lib/elasticsearch/nodes
117G    /var/lib/elasticsearch
94G     /var/lib/elasticsearch/nodes/0/indices/J9GUiOhkThilgzBp0i0RPQ/0
94G     /var/lib/elasticsearch/nodes/0/indices/J9GUiOhkThilgzBp0i0RPQ
93G     /var/lib/elasticsearch/nodes/0/indices/J9GUiOhkThilgzBp0i0RPQ/0/index

# date && curl -X GET "localhost:9200/_cat/indices/filebeat*?v"
Fri Jan 26 05:51:44 UTC 2024
health status index                              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   filebeat-8.11.1                    J9GUiOhkThilgzBp0i0RPQ   1   1  200504758            0       95gb           95gb
yellow open   filebeat-8.11.2                    hp8zHe1hTKmfuAJnTsy5fg   1   1   20445253            0      6.4gb          6.4gb
yellow open   filebeat-8.10.4                    HbQfL0pGS_ClJLLBWGFYUQ   1   1    2061898            0    488.6mb        488.6mb
yellow open   filebeat-7.17.15-2024.01.11        c_aY0AAiQjuJuDmFmCk8Hw   1   1       9924            0      5.6mb          5.6mb
yellow open   filebeat-8.11.1-2024.01.11         M_CDQqo9QXCjb6VeF0mk4A   1   1   19574081            0        9gb            9gb
yellow open   filebeat-8.10.4-2024.01.12         7ah5nAK6ShSf8oQhCSar3w   1   1      51352            0     10.2mb         10.2mb
yellow open   filebeat-8.11.1-2024.01.12         WhYy0DXBRe-85GB1e-ahWw   1   1   13762627            0      6.2gb          6.2gb
yellow open   filebeat-7.17.15-2024.01.12        S_Xu9N9wQXmIPogZl0a9Aw   1   1       7726            0      4.6mb          4.6mb
yellow open   filebeat-8.11.2-2024.01.12         ATocymT0SVy1sYqYmivaxg   1   1    1224750            0    405.2mb        405.2mb
yellow open   filebeat-8.11.2-2024.01.11         4CmGRMeURxGMyatRaPf-Wg   1   1    1763247            0    578.6mb        578.6mb
yellow open   filebeat-7.17.15-2024.01.22-000066 yUAzRzSCTLaElmzwt3UL5w   1   1      36925            0     12.7mb         12.7mb

# date && curl -X GET "localhost:9200/_cat/indices/*?v"
Fri Jan 26 06:00:02 UTC 2024
health status index                              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .reporting-2023-12-03              wPLklvtUSMqva6wIzY-DxQ   1   0          1            0      1.9mb          1.9mb
yellow open   filebeat-8.11.2-2024.01.12         ATocymT0SVy1sYqYmivaxg   1   1    1224750            0    405.2mb        405.2mb
green  open   .apm-agent-configuration           tI3nICpPQlWLzlPVFJPg3w   1   0          0            0       227b           227b
yellow open   filebeat-8.11.2-2024.01.11         4CmGRMeURxGMyatRaPf-Wg   1   1    1763247            0    578.6mb        578.6mb
yellow open   filebeat-7.17.15-2024.01.22-000066 yUAzRzSCTLaElmzwt3UL5w   1   1      36996            0     12.8mb         12.8mb
green  open   .tasks                             7c--yr7bTOe8ADz4vJ6ZsQ   1   0          6            0     41.1kb         41.1kb
yellow open   filebeat-8.11.1                    J9GUiOhkThilgzBp0i0RPQ   1   1  200617915            0       97gb           97gb
green  open   .geoip_databases                   4vJGhr5ySUWyUoNXYZxXbw   1   0         43           42     40.4mb         40.4mb
yellow open   filebeat-8.11.2                    hp8zHe1hTKmfuAJnTsy5fg   1   1   20445253            0      6.4gb          6.4gb
yellow open   filebeat-8.10.4                    HbQfL0pGS_ClJLLBWGFYUQ   1   1    2061898            0    488.7mb        488.7mb
yellow open   filebeat-7.17.15-2024.01.11        c_aY0AAiQjuJuDmFmCk8Hw   1   1       9924            0      5.6mb          5.6mb
green  open   .apm-custom-link                   rH28BebpTOKIA2KpKxR4aw   1   0          0            0       227b           227b
yellow open   filebeat-8.10.4-2024.01.12         7ah5nAK6ShSf8oQhCSar3w   1   1      51352            0     10.2mb         10.2mb
yellow open   filebeat-8.11.1-2024.01.11         M_CDQqo9QXCjb6VeF0mk4A   1   1   19574081            0        9gb            9gb
yellow open   filebeat-7.17.15-2024.01.12        S_Xu9N9wQXmIPogZl0a9Aw   1   1       7726            0      4.6mb          4.6mb
yellow open   filebeat-8.11.1-2024.01.12         WhYy0DXBRe-85GB1e-ahWw   1   1   13762627            0      6.2gb          6.2gb
green  open   .kibana_7.17.15_001                XT8tkEdKSASxfkVb5__Xig   1   0       2461           35        3mb            3mb
green  open   .async-search                      NjkQXv-hS2m8W7GzLjcbFA   1   0          0            0       258b           258b
green  open   .kibana_task_manager_7.17.15_001   XOJAZtH8TxCO_fsH-I0cqA   1   0         17       110959    101.6mb        101.6mb

GET /filebeat-7.17.15-2024.01.22-000066/_ilm/explain

{
  "indices" : {
    "filebeat-7.17.15-2024.01.22-000066" : {
      "index" : "filebeat-7.17.15-2024.01.22-000066",
      "managed" : true,
      "policy" : "filebeat",
      "lifecycle_date_millis" : 1705923192297,
      "age" : "3.75d",
      "phase" : "hot",
      "phase_time_millis" : 1705923192530,
      "action" : "rollover",
      "action_time_millis" : 1705923192530,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1705923192530,
      "phase_execution" : {
        "policy" : "filebeat",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "10gb",
              "max_primary_shard_size" : "10gb",
              "max_age" : "7d"
            }
          }
        },
        "version" : 13,
        "modified_date_in_millis" : 1705325399371
      }
    }
  }
}

GET /<filebeat-7.17.15>

{
  "filebeat-7.17.15-2024.01.22-000066" : {
    "aliases" : {
      "filebeat-7.17.15" : {
        "is_write_index" : true
      }

GET _cat/allocation?v&s=disk.indices&h=shards,disk.indices,disk.used,disk.available,disk.total,disk.percent

#! Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.
shards disk.indices disk.used disk.total disk.percent
    11                                               
    25      117.2gb     124gb      242gb           51

Thanks,

2

Hi,

This index is not managed by the ILM policy you have in place, which is why it's not being rolled over or deleted.

If you want to apply your ILM policy to this index, you can do so by updating the index settings.

PUT /filebeat-8.11.1/_settings
{
  "index.lifecycle.name": "filebeat"
}

Regards

3

Hello,

Thank you for the response. Will test and update here on the same thread.

Thanks,

4

Hello,

Please find the GET request details below for the filebeat-8.11.1

{
  "filebeat-8.11.1" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "filebeat-8.11.1",
        "creation_date" : "1705083736588",
        "number_of_replicas" : "1",
        "uuid" : "J9GUiOhkThilgzBp0i0RPQ",
        "version" : {
          "created" : "7171599"
        }
      }
    }
  }
}

Question: simply running the PUT request as mentioned above will fix the issue?

PUT /filebeat-8.11.1/_settings
{
  "index.lifecycle.name": "filebeat"
}

Because when ILM policy was implemented for filebeat-7.x before it was the below procedure which was followed since the logic was filebeat --> Logstash --> Elasticsearch

  1. Fix Logstash output
  2. Point filebeat output to elasticsearch
  3. ./filebeat setup -e (your command is not correct)
  4. Point filebeat output back to logstash
  5. Start Filebeat

The same procedure has to be run when applying the ILM for filebeat-8.11.1? Please suggest.

Also, as a note which I have informed in the previous forum discussions as below.

And 8.x Elastic Handles all this slightly different with Datastreams and if you are mizxing 7.x and 8.x Beats vs Logstash vs Elasticsearch you may have issues...

If you use 8.x Elasticsearch and Logstash you will need to use the output config shown on this page....

Thanks,
Ravi

6

Hello,

On my recent post along with the details and information shared. will running the below step alone will fix the issue? Please suggest

PUT /filebeat-8.11.1/_settings
{
  "index.lifecycle.name": "filebeat"
}

Thanks,

7

Hello,

Since this issue I am facing on the production, please can you confirm on my recent posts.

Thanks,
Ravi

8

Hi @yago82

Can you please confirm if this in not going to mess up with the existing ILM policy running on the live server? Because the ILM policy on the live server exists for filebeat version 7.17.15 and is in operational mode. Please find the details below.

GET /_ilm/policy/filebeat

{
  "filebeat" : {
    "version" : 13,
    "modified_date" : "2024-01-15T13:29:59.371Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "10gb",
              "max_primary_shard_size" : "10gb",
              "max_age" : "7d"
            }
          }
        },
        "delete" : {
          "min_age" : "3d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    },
    "in_use_by" : {
      "indices" : [
        "filebeat-7.17.15-2024.01.12",
        "filebeat-7.17.15-2024.02.05-000068",
        "filebeat-7.17.15-2024.01.29-000067"
      ],
      "data_streams" : [ ],
      "composable_templates" : [
        "filebeat-7.17.15"
      ]
    }
  }
}

GET _ilm/status

{
  "operation_mode" : "RUNNING"
}

Thanks,

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.