Kibana alerts (cluster health yellow) on new index creation

Elastic Stack Kibana
#1

(not sure if I should put this in elasticsearch or kibana topic)
Setup: logstash -> elasticsearch, index name is based on timestamp of event, so every day around midnight we get a new index.

The issue is twofold:

  • elasticsearch momentarily marks the cluster as yellow since the new index is on only one node
  • kibana notifies everyone about it because of the cluster health alert

Is there a way to tell either kibana (alerting) or elasticsearch not to worry when it's yellow for less than e.g. a minute? Or even better, to ignore new indices in cluster health for X time? Nothing in our stack breaks at less than a minute of downtime so it wouldn't be an issue, and I really want to avoid a "boy who cried wolf" scenario when anything important goes wrong.

#2

What kind of alert is this? I'd hope you could tweak the alert somehow, like, as you suggest, only fire if not green for X amount of time.

We have an issue open to allow muting via a schedule - https://github.com/elastic/kibana/issues/65706 - if the index is created at the same time every night, within a small window, such a capability would probably work for this situation. Feel free to post a comment to the issue if you have other thoughts on the capability.

#3

We're guessing you are probably using the Stack Monitoring Cluster alerts. It appears it doesn't handle your use case currently, but the suggestion is to open a feature request issue in our Kibana Github repository, and add a label Team: monitoring to it, so it gets directed in the right place.

#4

Thanks! That seems to be right. Is it possible that I can't add labels to it? I can't seem to find how to do that. For reference, this is the one: https://github.com/elastic/kibana/issues/82925

#5

Thanks for the issue, I added the label. I just realized that folks outside of Elastic probably can't add labels, sorry for suggesting otherwise.

#6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.