How to delete .monitoring*mb* indices automatically after period of time?

Elastic Stack Beats
,
1

Hi,

We are using Cloud Platinum subscription.
We enabled Metricbeat monitoring on 21st of January.

Seems like .monitoring indices (old and new) are deleted only after 1 month or not deleted at all.

Can we control retention period on mb monitoring system indices?
Are we supposed to set up ILM policies (and add it to system index templates) to control rollover and deletion of these system indices?

Please note that we have xpack.monitoring.history.duration set to "3d" which is not working. Is that related only to legacy monitoring and not to metribeat?

GET _cluster/settings?filter_path=*.xpack.monitoring&include_defaults=true

    {
      "persistent" : {
        "xpack" : {
          "monitoring" : {
            "collection" : {
              "enabled" : "true",
              "interval" : "10s"
            },
            "exporters" : {
              "__no-default-local__" : {
                "type" : "local",
                "enabled" : "false"
              }
            }
          }
        }
      },
      "defaults" : {
        "xpack" : {
          "monitoring" : {
            "collection" : {
              "cluster" : {
                "stats" : {
                  "timeout" : "10s"
                }
              },
              "node" : {
                "stats" : {
                  "timeout" : "10s"
                }
              },
              "indices" : [ ],
              "ccr" : {
                "stats" : {
                  "timeout" : "10s"
                }
              },
              "enrich" : {
                "stats" : {
                  "timeout" : "10s"
                }
              },
              "index" : {
                "stats" : {
                  "timeout" : "10s"
                },
                "recovery" : {
                  "active_only" : "false",
                  "timeout" : "10s"
                }
              },
              "ml" : {
                "job" : {
                  "stats" : {
                    "timeout" : "10s"
                  }
                }
              }
            },
            "history" : {
              "duration" : "3d"
            },
            "elasticsearch" : {
              "collection" : {
                "enabled" : "true"
              }
            },
            "enabled" : "true"
          }
        }
      }
    }

Any help is appreciated.

Kind regards,
Dane

2

I also have some questions about .ds-logs-elastic_agent.filebeat-default.* indices which are larger and occupy a Hot data node.
These system indices are by default set to "_tier_preference": "data_warm,data_hot".

  • Why are these indices located in Hot node when we also have Warm nodes available?
  • How are templates for these indices named?
  • And also the same question for them about retention period.

Thanks,
Dane

3

Welcome @dabo!

How many ES clusters are involved here? Is it a single one, or are you using a dedicated cluster for monitoring data?

4

Hi Chris,

Thanks! :slight_smile:

Currently we have only one cluster for all.

5

I think this is the problem. We have automated curation of the .monitoring-* indices through the xpack.monitoring.history.duration cluster setting, but it requires local exporter enabled.

Did you explicitly disable this local exporter? I'd expect it to be enabled by default

6

No, didn't change anything explicitly.

I was thinking the same if that should be enabled, but thought it is only for legacy monitoring, not for monitoring via Metricbeat...?

If we are not exporting data back to our cluster where is the data we see in Stack Monitoring coming from?

7

You're right, it's confusing. This will change soon, but for now, you'll need to enable the local exporter to kick off the curator task in Elasticsearch which respects the xpack.monitoring.history.duration cluster setting. Currently, ILM is not used to curate monitoring indices.

1 Like
8

Great, thanks! We will try that.

Will that setting affect the .ds-* indices as well? These are having must greater impact on our disk at the moment.

UPDATE: found an answer to all my questions about .ds-* indices.

Thank you for your help, you can close the topic.

Kind regards,
Dane

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.