.monitoring-es indexes blow up, have no policies associated?

[wrobitza_aveq (Werner Robitza)]

I've enabled Stack Monitoring in Kibana, just for the purpose of trying it out. Now, after a couple of days, I see that my stack is using a lot of disk memory.

I was able to delete the individual .es-monitoring* indices that used up this space, but I would like to modify the policy that creates them in the first place, so that they are only kept for one day or 1GB at most.

What I see from this topic is not really helpful; it says I can safely delete these indices. But I don't want to do that manually. I also notice that for APM, for instance, there are index lifecycle policies that I can modify, but for monitoring, I don't see any.

What can I do to prevent these large indices?

[stephenb (Stephen Brown)]

HI @wrobitza_aveq Welcome to the community

@wrobitza_aveq

I've enabled Stack Monitoring in Kibana, just for the purpose of trying it out. Now, after a couple of days, I see that my stack is using a lot of disk memory.

Is this Self Managed or in Elastic Cloud

If Self Managed How did you do enable monitoring / Which Method? Metricbeat / Self Monitoring?

And how did you determine that ILM is not enabled for those indices? (just validating)

[wrobitza_aveq (Werner Robitza)]

This is self-managed. I enabled it by going to Management » Stack Monitoring and following the GUI wizard there.

There are no ILM policies visible when I click on the index in Index Management. Other indices like the ones from APM (e.g., traces) show the ILM policies.

[stephenb (Stephen Brown)]

@wrobitza_aveq

This is self-managed. I enabled it by going to Management » Stack Monitoring and following the GUI wizard there.

That is the issue... "Self Monitoring" Will be deprecated at some point / soon (this is part of the reason why ), it is just the "quick" way to set up monitoring.

If you had used metricbeat etc... then there would be policies... although metricbeat take a bit more work but then it is like any other index and it will have an ILM policy etc.

There is no simple why to add ILM to those indices...

[wrobitza_aveq (Werner Robitza)]

That's good to know. I didn't notice that when enabling the feature a week ago. Maybe the deprectation message needs to be more prominent?

In any case, I ran:

PUT /_cluster/settings
{
  "persistent": {
    "xpack.monitoring.elasticsearch.collection.enabled": false
  }
}

and:

PUT /_cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": false
  }
}

and stopped monitoring for now.

[stephenb (Stephen Brown)]

Yes perhaps... I think it is in the GUI and the Docs pretty prominently.

Not sure what version you are on but I got the deprecation warning with the setting

PUT /_cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}


#! [xpack.monitoring.collection.enabled] setting was deprecated in Elasticsearch and will be removed in a future release.
{
  "acknowledged": true,
  "persistent": {
    "xpack": {
      "monitoring": {
        "collection": {
          "enabled": "true"
        }
      }
    }
  },
  "transient": {}
}

[wrobitza_aveq (Werner Robitza)]

Yes perhaps... I think it is in the GUI and the Docs pretty prominently.

Indeed. In contrast to Legacy APM (standalone server) though, the ILM policies are not present, which caused the whole confusion from my side.

And I did get the same messages on 8.3.3 running these commands!

So I guess I will check out Metricbeat in the future. Thanks for your quick help!