Internal monitoring and log indices have "live forever" ILM policies

Elastic Stack Elasticsearch
,
1

Hi!
I was investigating an issue with too much retained data in our ES cloud cluster and realized that a lot of it comes from the internal monitoring and log indices. Specifically, the following indices with corresponding ILM policies:

  • .ds-metricbeat-* indices --> "metricbeat" ILM policy

  • .ds-elastic-cloud-logs-* --> "elastic-cloud-logs" ILM policy

Here is one of the policies - the other one is the same:

{
  "elastic-cloud-logs": {
    "version": 1,
    "modified_date": "2022-11-08T16:39:52.102Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_primary_shard_size": "50gb",
              "max_age": "30d"
            }
          }
        }
      }
    },
    "in_use_by": {
      "indices": [
        ".ds-elastic-cloud-logs-8-2023.02.08-000005",
        ".ds-elastic-cloud-logs-8-2023.03.10-000006",
        ".ds-elastic-cloud-logs-8-2023.04.09-000007",
        ".ds-elastic-cloud-logs-8-2023.01.09-000004",
        ".ds-elastic-cloud-logs-8-2023.01.07-000003"
      ],
      "data_streams": [
        "elastic-cloud-logs-8"
      ],
      "composable_templates": [
        "elastic-cloud-logs-8"
      ]
    }
  }
}

I know I can change them manually (or add to our automation) however I like, but:
a few questions:

  1. is this OK/advisable to change internal ES ILM policies?
  2. and if I do that - would they be, potentially ,updated when the managed ES cluster is updated? (for example, reverted back tot he default version)?
  3. and related to above - DO ES cluster upgrade automatically for minor versions? or is it a manually "kicked off" process only?

Thank you!
Marina

2

Hi @ppine7

3

Great, thank you, @stephenb !

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.