(not sure if I should put this in elasticsearch or kibana topic)
Setup: logstash -> elasticsearch, index name is based on timestamp of event, so every day around midnight we get a new index.
The issue is twofold:
elasticsearch momentarily marks the cluster as yellow since the new index is on only one node
kibana notifies everyone about it because of the cluster health alert
Is there a way to tell either kibana (alerting) or elasticsearch not to worry when it's yellow for less than e.g. a minute? Or even better, to ignore new indices in cluster health for X time? Nothing in our stack breaks at less than a minute of downtime so it wouldn't be an issue, and I really want to avoid a "boy who cried wolf" scenario when anything important goes wrong.
What kind of alert is this? I'd hope you could tweak the alert somehow, like, as you suggest, only fire if not green for X amount of time.
We have an issue open to allow muting via a schedule - https://github.com/elastic/kibana/issues/65706 - if the index is created at the same time every night, within a small window, such a capability would probably work for this situation. Feel free to post a comment to the issue if you have other thoughts on the capability.