Kibana alerts (cluster health yellow) on new index creation

(not sure if I should put this in elasticsearch or kibana topic)
Setup: logstash -> elasticsearch, index name is based on timestamp of event, so every day around midnight we get a new index.

The issue is twofold:

  • elasticsearch momentarily marks the cluster as yellow since the new index is on only one node
  • kibana notifies everyone about it because of the cluster health alert

Is there a way to tell either kibana (alerting) or elasticsearch not to worry when it's yellow for less than e.g. a minute? Or even better, to ignore new indices in cluster health for X time? Nothing in our stack breaks at less than a minute of downtime so it wouldn't be an issue, and I really want to avoid a "boy who cried wolf" scenario when anything important goes wrong.

What kind of alert is this? I'd hope you could tweak the alert somehow, like, as you suggest, only fire if not green for X amount of time.

We have an issue open to allow muting via a schedule - https://github.com/elastic/kibana/issues/65706 - if the index is created at the same time every night, within a small window, such a capability would probably work for this situation. Feel free to post a comment to the issue if you have other thoughts on the capability.

We're guessing you are probably using the Stack Monitoring Cluster alerts. It appears it doesn't handle your use case currently, but the suggestion is to open a feature request issue in our Kibana Github repository, and add a label Team: monitoring to it, so it gets directed in the right place.

Thanks! That seems to be right. Is it possible that I can't add labels to it? I can't seem to find how to do that. For reference, this is the one: https://github.com/elastic/kibana/issues/82925

Thanks for the issue, I added the label. I just realized that folks outside of Elastic probably can't add labels, sorry for suggesting otherwise.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.