I am trying to define Monitoring rule to detect pod status=failed.
I've used Discover and the parameters below:
Data view: pod-status-ad-hoc
Filter: metrics.k8s.pod.phase: exists
Query: ( ( "failed": * ) AND ( "resource.attributes.k8s.pod.name": * ) )
It returns 3 pods in failed status.
If I go to Alerts > Create search threshold rule, it opens the ‘Create Rule’ for Rule type = Elasticsearch query, populates data view, query and filter.
As a result, it will alert for the number of entries for pod 'failed'.
Instead of counting the number of documents, what I’d like to do is generate an alert for each pod that is in ‘failed’ status.
I could not find a way to implement this type of monitoring rule. Any suggestion? Thanks in advance.