Watcher - Webhook action iterate through aggregated results and show it in json format

Elastic Stack Elasticsearch
1

I am creating a watch which queries and aggregates the output. How do I loop through the aggregated results and create a json format and send through web hook. Tried various solutions but still stuck it. Any help will be appreciated.

PUT _xpack/watcher/watch/iad_watch
{
"trigger" : { "schedule" : { "interval" : "60s" }},
"input" : {
"search" : {
"request" : {
"body" : {
"query" : {
"bool" : {
"must" : [

                             {"match" : {"type":"iad" }},
                             {"match" : {"priority":"ERROR" }}
                       ],
                       "filter" : {
                           "range" : {
                                    "@timestamp" : {
                                           "gte" : "now-1d",
                                           "lt" :  "now"
                                    }
                           }
                       },
            ,
                  "size" :0,
                  "aggs": {
                    "analyst_name": {
                      "terms": {
                         "field":   "iAnalyst.keyword"   
                        },
                    "aggs": {
                      "group_docs": {
                        "top_hits": {
                            "size": 1,
                            "sort": [
                            {
                              "@timestamp": {
                                "order": "desc"
                                }
                            }
                          ]
                        }
                    }
                  }
                }
            }
                     
             }
          }
      }
  }
}

},
"condition" : {
"compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}
},
"actions" : {
"iad_webhook" : {
"webhook" : {
"auth" : {
"basic" : {
"username" : "admin",
"password" : "admin"
}
},
"method" : "POST",
"host" : "172.18.187.151",
"port" : 8161,
"path" : "/api/message",
"params" : {
"destination" : "topic://TOPIC.IAD.WATCHER.ERRORS"
},
"headers" : {
"Content-Type" : "application/json"
},
"body" : "{ "total_errors" : {{ctx.payload.hits.total}}, "iAnalyst" : "{{ctx.payload.aggregate.analyst_name.hits.hits.0._source.iAnalyst}}", "iAnalystHost" : "{{ctx.payload.aggregate.analyst_name.hits.hits.0._source.host}}", "last_error_message" : "{{ctx.payload.aggregate.analyst_name.hits.hits.0._source.message}}", "error_datetime" : "{{ctx.payload.aggregate.analyst_name.hits.hits.0._source.@timestamp}}" }"

       }
  }

}}

2

Hey,

please take your time to properly format your messages in markdown. This makes it real hard to read.

You can loop through an list of items like this

{{#ctx.payload.aggregate.analyst_name.hits.hits}}
Host: {{_source.host}} Message: {{_source.message}}
{{/ctx.payload.aggregate.analyst_name.hits.hits.}}

--Alex

3

Thanks Alexander Reelsen. Sorry about formatting. Are there tags to wrap the code.

I tried the recommendations as below. Not sure what I am missing, but I dont see any results

"actions" : {
"log" : {
"logging" : {
"text" : "Encountered: {{ctx.payload.hits.total}} in the last hour on {{#ctx.payload.aggregations.analyst_name.hits.hits}}{{_source.host}}: {{_source.message}}, {{/ctx.payload.aggregations.analyst_name.hits.hits}}"

       }
  }

}

Below is the ouput of the search query I sent in my original post. I am trying to parse results from aggregations tags.

{
"took": 17,
"timed_out": false,
"_shards": {
},
"hits": {
"total": 67,
"max_score": 0,
"hits": []
},
"aggregations": {
"analyst_name": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "mockb-1",
"doc_count": 66,
"group_docs": {
"hits": {
"total": 66,
"max_score": null,
"hits": [
{
"_index": "iad-2017.02.08",
"_type": "iad",
"_id": "AVob5Fd0epr-8jzLv2J3",
"_score": null,
"_source": {
"method": "?",
"thread": "analystDesktop-5637-MacBook-Pro.local6dd6-1",
"message": "I am testing IAD error logging to logstash",
"priority": "ERROR",
"type": "iad",
"path": "net.interactions.ianalyst.jms.IADMessageReceiver",
"@timestamp": "2017-02-08T04:03:53.833Z",
"file": "?:?",
"iAnalyst": "mockb-1",
"@version": "1",
"host": "172.18.187.85:57938",
"logger_name": "net.interactions.ianalyst.jms.IADMessageReceiver",
"class": "?",
"timestamp": 1486526633765
},
"sort": [
1486526633833
]
}
]
}
}
},
{
"key": "mockb-3",
"doc_count": 1,
"group_docs": {
"hits": {
"total": 1,
"max_score": null,
"hits": [
{
"_index": "iad-2017.02.07",
"_type": "iad",
"_id": "AVol6L7Repr-8jzL094Y",
"_score": null,
"_source": {
"message": "I am testing IAD error logging to logstash",
"priority": "ERROR",
"type": "iad",
"@timestamp": "2017-02-08T21:03:53.833Z",
"iAnalyst": "mockb-3",
"host": "172.18.187.85:57938"
},
"sort": [
1486587833833
]
}
]
}
}
}
]
}
}
}

4

You can use common markdown to format your posts.

5

Also, please take your time and compare your script example with the path of the json data, you will find the buckets field missing for example.

6

Thanks Alexander. That was useful.
Future reference on how to iterate the nested arrays in watcher using mustang

"{ "total_errors" : {{ctx.payload.hits.total}},{{#ctx.payload.aggregations.analyst_name.buckets}}{{#group_docs.hits.hits}} {"iAnalyst" : "{{_source.iAnalyst}}", "iAnalystHost" : "{{_source.host}}", "last_error_message" : "{{_source.message}}", "error_datetime" : "{{_source.@timestamp}}"}, {{/group_docs.hits.hits}}{{/ctx.payload.aggregations.analyst_name.buckets}}"

7

Thanks, this was really helpful for me too. Is there a way to show all hits in the alert rather than just 10?

8

please open new threads instead of appending to older ones.

You may want to check out size parameter of the terms aggregation

--Alex

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.