# Generate multiple alerts from elastic query

**URL:** <https://discuss.elastic.co/t/generate-multiple-alerts-from-elastic-query/351113>\
**Category:** Kibana\
**Tags:** elastic-stack-alerting\
**Created:** [January 16, 2024, 1:04am UTC](https://discuss.elastic.co/t/generate-multiple-alerts-from-elastic-query/351113 "2024-01-16T01:04:55Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![umesh2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/umesh2020/32/126038_2.png) [@umesh2020](https://discuss.elastic.co/u/umesh2020)\
**Post date:** [January 16, 2024, 1:04am UTC](https://discuss.elastic.co/t/generate-multiple-alerts-from-elastic-query/351113/1 "2024-01-16T01:04:55Z")

</div>

I want to generate an alert whenever a pod is restarted in kubernetes. I am planning to use kubernetes.container.status.restarts to identify whether pods are restarted or not. I would like to have a single elasticsearch query and generate multiple alerts one per pod per namespace which is restarted. I know this is a fairly standard requirement, but couldn't find any elasticsearch query examples. I tried the following query for a specific pod, but I am unable to generate individual alerts, but only a single alert. Can you please help ? Here is my elasticsearch alert query:

```auto
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "kubernetes.container.name": "hostinterface"
          }
        },
        {
          "range": {
            "kubernetes.container.status.restarts": {
              "gt": 1
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "group_by_namespace": {
      "terms": {
        "field": "kubernetes.namespace",
        "size": 10
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![yago82](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yago82/32/97755_2.png) [@yago82](https://discuss.elastic.co/u/yago82)\
**Post date:** [January 16, 2024, 1:30pm UTC](https://discuss.elastic.co/t/generate-multiple-alerts-from-elastic-query/351113/2 "2024-01-16T13:30:35Z")

</div>

> [@umesh2020](#):
>
> I want to generate an alert whenever a pod is restarted in kubernetes. I am planning to use kubernetes.container.status.restarts to identify whether pods are restarted or not. I would like to have a single elasticsearch query and generate multiple alerts one per pod per namespace which is restarted. I know this is a fairly standard requirement, but couldn't find any elasticsearch query examples. I tried the following query for a specific pod, but I am unable to generate individual alerts, but only a single alert. Can you please help ? Here is my elasticsearch alert query:
> 
> ```auto
> {
> "query": {
> "bool": {
> "must": [
> {
> "match": {
> "kubernetes.container.name": "hostinterface"
> }
> },
> {
> "range": {
> "kubernetes.container.status.restarts": {
> "gt": 1
> }
> }
> }
> ]
> }
> },
> "aggs": {
> "group_by_namespace": {
> "terms": {
> "field": "kubernetes.namespace",
> "size": 10
> }
> }
> }
> }
> 
> ```

Hi,

it seems like you're missing a sub-aggregation to group by pod name within each namespace. Try with this:

```auto
"aggs": {
    "group_by_namespace": {
      "terms": {
        "field": "kubernetes.namespace",
        "size": 10
      },
      "aggs": {
        "group_by_pod": {
          "terms": {
            "field": "kubernetes.pod.name",
            "size": 10
          }
        }
      }
    }
  }

```

Regards

---

<div class="post-metadata">

**Author:** ![umesh2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/umesh2020/32/126038_2.png) [@umesh2020](https://discuss.elastic.co/u/umesh2020)\
**Post date:** [January 16, 2024, 3:57pm UTC](https://discuss.elastic.co/t/generate-multiple-alerts-from-elastic-query/351113/3 "2024-01-16T15:57:35Z")

</div>

Thank you. I tried your suggestion. However, I am not getting multiple alerts, but only a single alert. Can you let me know if there is any issue in my elastic altert ? Here is a screenshot for your reference

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/4/f4282fd9c6e4554de7efe25ac1df127ed87847f2.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 16, 2024, 4:18pm UTC](https://discuss.elastic.co/t/generate-multiple-alerts-from-elastic-query/351113/4 "2024-01-16T16:18:43Z")

</div>

Hi @umesh2020

I think you can do this fairly simply with a normal metric threshold alert and then group by namespace ...

The Pattern you're following is one of the base use cases for metric threshold alert

> **[Create a metrics threshold rule | Elastic Observability \[8.11\] | Elastic](https://www.elastic.co/guide/en/observability/current/metrics-threshold-alert.html#metrics-threshold-alert)**

---

<div class="post-metadata">

**Author:** ![umesh2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/umesh2020/32/126038_2.png) [@umesh2020](https://discuss.elastic.co/u/umesh2020)\
**Post date:** [January 16, 2024, 4:42pm UTC](https://discuss.elastic.co/t/generate-multiple-alerts-from-elastic-query/351113/5 "2024-01-16T16:42:27Z")

</div>

Thank you for your response. The reason I chose to use query is because with metric threshold, I was unable to get pod that restarted, the namespace etc. Will try it out and get back if I have further questions

---

<div class="post-metadata">

**Author:** ![umesh2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/umesh2020/32/126038_2.png) [@umesh2020](https://discuss.elastic.co/u/umesh2020)\
**Post date:** [January 16, 2024, 5:12pm UTC](https://discuss.elastic.co/t/generate-multiple-alerts-from-elastic-query/351113/6 "2024-01-16T17:12:32Z")

</div>

I created a metric threshold rule as you suggested, but the issue is I don't get the pod name which restarted. Here is a partial context for your reference.

```auto
{
  "alertState": "ALERT",
  "group": "fi1-https",
  "groupByKeys": {
    "kubernetes": {
      "namespace": "fi1-https"
    }
  },
  "metric": {
    "condition0": "kubernetes.container.status.restarts"
  },
  "reason": "kubernetes.container.status.restarts is 187 in the last 1 min for fi1-https. Alert when > 1.",
  "threshold": {
    "condition0": [
      "1"
    ]
  },
  "timestamp": "2024-01-16T17:05:56.382Z",
  "value": {
    "condition0": "187"
  },
  "tags": []
}

```

Is there a way I can get the pod name and also the query results as part of the context ?

---

<div class="post-metadata">

**Author:** ![umesh2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/umesh2020/32/126038_2.png) [@umesh2020](https://discuss.elastic.co/u/umesh2020)\
**Post date:** [January 16, 2024, 5:34pm UTC](https://discuss.elastic.co/t/generate-multiple-alerts-from-elastic-query/351113/7 "2024-01-16T17:34:06Z")

</div>

I got the pod name by grouping over both namespace and pod name. However, let me know if there is a way to get a link to the query results which I can use to get more details about the event. Appreciate your help.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 16, 2024, 7:27pm UTC](https://discuss.elastic.co/t/generate-multiple-alerts-from-elastic-query/351113/8 "2024-01-16T19:27:39Z")

</div>

> [@umesh2020](#):
>
> let me know if there is a way to get a link to the query results which I can use to get more details about the event.

Kibana Alerts do not work that way...

what I do is just put `{{.}}` in your action message action and you should see everthing that is available to you...

Then you can use what you want...

Give that a try...

---

<div class="post-metadata">

**Author:** ![umesh2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/umesh2020/32/126038_2.png) [@umesh2020](https://discuss.elastic.co/u/umesh2020)\
**Post date:** [January 18, 2024, 2:25pm UTC](https://discuss.elastic.co/t/generate-multiple-alerts-from-elastic-query/351113/9 "2024-01-18T14:25:16Z")

</div>

Thank you. This is a very good tip

---

<div class="post-metadata">

**Author:** ![umesh2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/umesh2020/32/126038_2.png) [@umesh2020](https://discuss.elastic.co/u/umesh2020)\
**Post date:** [January 18, 2024, 2:50pm UTC](https://discuss.elastic.co/t/generate-multiple-alerts-from-elastic-query/351113/10 "2024-01-18T14:50:39Z")

</div>

I have one more small issue. I am creating the rule using the following query using Kibana REST API. The query get's created successfully, however the query behaves like kuberntes.container.name: \* instead of filtering out the containers that I am interested. I have included a snippet of my parameters for your reference.

```auto
        "name": "pod_status_restart",
        "notify_when": "onActionGroupChange",
        "params": {
            "alertOnGroupDisappear": true,
            "alertOnNoData": true,
            "criteria": [
                {
                    "aggType": "avg",
                    "comparator": ">",
                    "metric": "kubernetes.container.status.restarts",
                    "threshold": [
                        3
                    ],
                    "timeSize": 1,
                    "timeUnit": "m"
                }
            ],
            "filterQueryText": "kubernetes.container.name: hostinterface OR kubernetes.container.name: keycloak OR kubernetes.container.name: nginx OR kubernetes.container.name: redis",
            "groupBy": [
                "kubernetes.namespace",
                "kubernetes.pod.name"
            ],
            "nodeType": "pod",
            "sourceId": "default"
        },
        "revision": 0,
        "rule_type_id": "metrics.alert.threshold",

```

However, if I go to Kibana UI and then save the query again, it starts working. The difference I see is that after I save the rule, it converts the filterQueryText to an actual query and saves it in "filterQuery" field. Can you let me know why using filterQueryText doesn't work ?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 18, 2024, 3:03pm UTC](https://discuss.elastic.co/t/generate-multiple-alerts-from-elastic-query/351113/11 "2024-01-18T15:03:19Z")

</div>

I Do not exactly recall but perhaps take a look at this thread It may help.

> [@Metric Threshold Alert reporting incorrect document count](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/33):
>
> Hey @stephenb, yes it seems you have a pretty good understanding of the situation. The alerts created through the metric explorer were all working as intended. However, we create our rules through the kibana API due to an integration with our react app. Looking at the rule from the UI, it looked alright since it had been showing the filterQueryText. Though underneath, it was missing a filterQuery value so in reality it wasn't filtering at all. I would think that the filterQuery would be derived…

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 15, 2024, 3:04pm UTC](https://discuss.elastic.co/t/generate-multiple-alerts-from-elastic-query/351113/12 "2024-02-15T15:04:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.