# DSL query in Kibana Rules does not work the same as from Dev Tools

**URL:** <https://discuss.elastic.co/t/dsl-query-in-kibana-rules-does-not-work-the-same-as-from-dev-tools/327577>\
**Category:** Kibana\
**Tags:** elastic-stack-alerting\
**Created:** [March 13, 2023, 3:27pm UTC](https://discuss.elastic.co/t/dsl-query-in-kibana-rules-does-not-work-the-same-as-from-dev-tools/327577 "2023-03-13T15:27:26Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![mohsin106](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohsin106/32/65203_2.png) [@mohsin106](https://discuss.elastic.co/u/mohsin106)\
**Post date:** [March 13, 2023, 3:27pm UTC](https://discuss.elastic.co/t/dsl-query-in-kibana-rules-does-not-work-the-same-as-from-dev-tools/327577/1 "2023-03-13T15:27:26Z")

</div>

Hello,

I'm using Elastic 7.17.6 and I have an alert rule set up which is using the below DSL query to search for the data:

```auto
{
  "size": 0,
  "query": {
    "bool": {
      "must": [
        { "match": { "empty": "true" }},
        { "match": { "type": "LINECARD" }}
      ]
    }
  },
  "aggs": {
    "unique_sources": {
      "filter": {
        "bool": {
          "must": [
            { "match": { "empty": "true" }},
            { "match": { "type": "LINECARD" }}
          ]
        }
      },
      "aggs": {
        "sources": {
          "terms": {
            "field": "source.keyword",
            "size": 10
          },
          "aggs": {
            "top_source": {
              "top_hits": {
                "size": 1
              }
            }
          }
        }
      }
    }
  }
}

```

I'm trying to only get one unique data set returned based on `source.keyword`.  
When I click on "Test Rule" I get this result `Query matched 6 documents in the last 1m.`  
I'm streaming data in from a network device so I do get duplicate data sets sent within a minute and I would like to receive a unique data set based on `source.keyword`.

Then in the "Message" section of the rule, I have this handlebar code to print the data from the above search result:

```auto
Alert
=
Date: {{context.date}}

Hardware removal was detected from below device:

{{ #context.hits }}
source: {{_source.source}}

slot: {{_source.slot}}

description: {{_source.description}}

type: {{_source.type}}

parent: {{_source.parent}}

slot empty: {{_source.empty}}

power status: {{_source.power_admin_state}}

{{/context.hits}}

```

I'm getting an email that prints all 6 documents that the DSL query matched. These 6 documents are duplicates. All six documents are from the same source. I would like the email to only return 1 unique data set. I hope that makes sense.

I don't know if there is a way in the handlebar code to parse the `#context.hits` to only return unique data? Or if the DSL query can be edited to accomplish this?

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [March 13, 2023, 4:25pm UTC](https://discuss.elastic.co/t/dsl-query-in-kibana-rules-does-not-work-the-same-as-from-dev-tools/327577/2 "2023-03-13T16:25:29Z")

</div>

I think your problem is that Kibana Rules does not yet fully support Elasticsearch aggregations (follow the GitHub issue here: [Elasticsearch Query Stack Alert Aggregation Support · Issue #95161 · elastic/kibana · GitHub](https://github.com/elastic/kibana/issues/95161))

In the meantime, you could accomplish this with [Watcher](https://www.elastic.co/guide/en/kibana/current/watcher-ui.html).

---

<div class="post-metadata">

**Author:** ![mohsin106](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohsin106/32/65203_2.png) [@mohsin106](https://discuss.elastic.co/u/mohsin106)\
**Post date:** [March 13, 2023, 5:53pm UTC](https://discuss.elastic.co/t/dsl-query-in-kibana-rules-does-not-work-the-same-as-from-dev-tools/327577/3 "2023-03-13T17:53:09Z")

</div>

Thanks for the info. I'll look into Watcher.

---

<div class="post-metadata">

**Author:** ![mohsin106](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohsin106/32/65203_2.png) [@mohsin106](https://discuss.elastic.co/u/mohsin106)\
**Post date:** [March 13, 2023, 7:40pm UTC](https://discuss.elastic.co/t/dsl-query-in-kibana-rules-does-not-work-the-same-as-from-dev-tools/327577/4 "2023-03-13T19:40:27Z")

</div>

I don't think Watcher works with boolean fields. I'm trying to create a watch and my options are:

- count()
- avg()
- sum()
- min()
- max()

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [March 14, 2023, 1:18pm UTC](https://discuss.elastic.co/t/dsl-query-in-kibana-rules-does-not-work-the-same-as-from-dev-tools/327577/5 "2023-03-14T13:18:28Z")

</div>

watcher allows for any arbitrary elasticsearch query or aggregation. You can then evaluate the result of that query using the `script` type of the `condition` to do anything you want.

See example here: [Watcher condition context | Painless Scripting Language [8.6] | Elastic](https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-watcher-condition-context.html)

---

<div class="post-metadata">

**Author:** ![mohsin106](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohsin106/32/65203_2.png) [@mohsin106](https://discuss.elastic.co/u/mohsin106)\
**Post date:** [March 17, 2023, 10:31pm UTC](https://discuss.elastic.co/t/dsl-query-in-kibana-rules-does-not-work-the-same-as-from-dev-tools/327577/6 "2023-03-17T22:31:03Z")

</div>

I'm able to get Watcher to fire when I run simulate. How can I iterate the "hits" in my email?

```auto
"actions": {
    "email_1": {
      "email": {
        "profile": "standard",
        "to": [
          "myeamil@email.com"
        ],
        "subject": "Watch [{{ctx.metadata.name}}] has exceeded the threshold",
        "body": {
          "text": "{{ #ctx.payload.hits}} ???? {{ /ctx.payload.hits }}"
        }
      }
    }
  }
}

```

The handlebar code from my Rules is not working here.

---

<div class="post-metadata">

**Author:** ![mohsin106](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohsin106/32/65203_2.png) [@mohsin106](https://discuss.elastic.co/u/mohsin106)\
**Post date:** [March 17, 2023, 11:03pm UTC](https://discuss.elastic.co/t/dsl-query-in-kibana-rules-does-not-work-the-same-as-from-dev-tools/327577/7 "2023-03-17T23:03:49Z")

</div>

If I print just `{{ ctx }}` in the email body I get the following response:

> **Click Me**
>
> ```auto
> {
> metadata = {
> name = Test Lab Watch,
> xpack = {
> type = json
> }
> }, watch_id = _inlined_, payload = {
> _shards = {
> total = 1,
> failed = 0,
> successful = 1,
> skipped = 0
> },
> hits = {
> hits = [],
> total = 6,
> max_score = null
> },
> took = 1,
> timed_out = false,
> aggregations = {
> unique_sources = {
> doc_count = 6,
> sources = {
> doc_count_error_upper_bound = 0,
> sum_other_doc_count = 0,
> buckets = [{
> doc_count = 6,
> top_source = {
> hits = {
> hits = [{
> _index = Test - components - 2023.03 .17,
> _type = _doc,
> _source = {
> part - no = ,
> parent = SLOT - 1,
> hardware - version = ,
> oper - status = INACTIVE,
> mfg - name = Test,
> description = ,
> serial - no = ,
> slot = LM - 1,
> source = 10.10 .10 .1,
> type = LINECARD,
> tid = site1,
> empty = true,
> power - admin - state = POWER_DISABLED,
> @timestamp = 2023 - 03 - 17 T22: 55: 34.618 Z,
> host = lab - agent,
> @version = 1,
> name = components
> },
> _id = Qe7J8YYB6ZJARmddHeVJ,
> _score = 6.320991
> }],
> total = 6,
> max_score = 6.320991
> }
> },
> key = 10.10 .10 .1
> }]
> },
> meta = {}
> }
> }
> }, id = _inlined__6d3b8a0f - c477 - 4259 - 811 d - 4e086350 f191 - 2023 - 03 - 17 T22: 55: 40.412949059 Z, trigger = {
> triggered_time = 2023 - 03 - 17 T22: 55: 40.412937978 Z,
> scheduled_time = 2023 - 03 - 17 T22: 55: 40.412937978 Z
> }, vars = {}, execution_time = 2023 - 03 - 17 T22: 55: 40.412949059 Z
> }
> 
> ```

I can print `{{ ctx.metadata }}` but can't figure out how to get into the `hits` array.

How can I iterate `{{ ctx }}` and print the `source`, `slot` and `empty` fields?

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [March 20, 2023, 8:00pm UTC](https://discuss.elastic.co/t/dsl-query-in-kibana-rules-does-not-work-the-same-as-from-dev-tools/327577/8 "2023-03-20T20:00:42Z")

</div>

Hi - nice job getting most of your watch to work.

There's a clue to what you need to do when you look at the contents of `ctx`. Notice that first of all, you need to reference the things by their hierarchy in the `ctx` object. So, really, `ctx.payload.aggregations.unque_sources.sources.buckets` is the array that you'll need to loop through (not `ctx.payload.hits`).

So, something like:

```auto
{{#ctx.payload.aggregations.airlines.buckets}}
IP={{key}} in slot {{top_source.hits.hits.0.slot}} had the following problem: {{top_source.hits.hits.0.description}} 
{{/ctx.payload.aggregations.airlines.buckets}}

```

Give it a shot!

---

<div class="post-metadata">

**Author:** ![mohsin106](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohsin106/32/65203_2.png) [@mohsin106](https://discuss.elastic.co/u/mohsin106)\
**Post date:** [March 22, 2023, 2:49pm UTC](https://discuss.elastic.co/t/dsl-query-in-kibana-rules-does-not-work-the-same-as-from-dev-tools/327577/9 "2023-03-22T14:49:55Z")

</div>

Thanks for the guidance. I was able to get it working following your instructions. Here is what it looks like now:

```auto
{{#ctx.payload.aggregations.unique_sources.sources.buckets}}
<div><b>Source</b>:{{key}}<br>
  {{#top_source.hits.hits}}
    <b>Description</b>: {{_source.description}}<br>
    <b>TID</b>: {{_source.tid}}<br>
    <b>Slot</b>: {{_source.slot}}<br>
    <b>Parent</b>: {{_source.parent}}<br>
    <b>Slot empty</b>: {{_source.empty}}<br>
    <b>Power Status</b>: {{_source.power_admin_state}}<br><br></div>
  {{/top_source.hits.hits}}
{{/ctx.payload.aggregations.unique_sources.sources.buckets}}

```

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [March 22, 2023, 4:30pm UTC](https://discuss.elastic.co/t/dsl-query-in-kibana-rules-does-not-work-the-same-as-from-dev-tools/327577/10 "2023-03-22T16:30:12Z")

</div>

nice!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 19, 2023, 4:30pm UTC](https://discuss.elastic.co/t/dsl-query-in-kibana-rules-does-not-work-the-same-as-from-dev-tools/327577/11 "2023-04-19T16:30:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.