Watcher - Webhook action iterate through aggregated results and show it in json format

I am creating a watch which queries and aggregates the output. How do I loop through the aggregated results and create a json format and send through web hook. Tried various solutions but still stuck it. Any help will be appreciated.

PUT _xpack/watcher/watch/iad_watch
{
"trigger" : { "schedule" : { "interval" : "60s" }},
"input" : {
"search" : {
"request" : {
"body" : {
"query" : {
"bool" : {
"must" : [

                             {"match" : {"type":"iad" }},
                             {"match" : {"priority":"ERROR" }}
                       ],
                       "filter" : {
                           "range" : {
                                    "@timestamp" : {
                                           "gte" : "now-1d",
                                           "lt" :  "now"
                                    }
                           }
                       },
            ,
                  "size" :0,
                  "aggs": {
                    "analyst_name": {
                      "terms": {
                         "field":   "iAnalyst.keyword"   
                        },
                    "aggs": {
                      "group_docs": {
                        "top_hits": {
                            "size": 1,
                            "sort": [
                            {
                              "@timestamp": {
                                "order": "desc"
                                }
                            }
                          ]
                        }
                    }
                  }
                }
            }
                     
             }
          }
      }
  }
}

},
"condition" : {
"compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}
},
"actions" : {
"iad_webhook" : {
"webhook" : {
"auth" : {
"basic" : {
"username" : "admin",
"password" : "admin"
}
},
"method" : "POST",
"host" : "172.18.187.151",
"port" : 8161,
"path" : "/api/message",
"params" : {
"destination" : "topic://TOPIC.IAD.WATCHER.ERRORS"
},
"headers" : {
"Content-Type" : "application/json"
},
"body" : "{ "total_errors" : {{ctx.payload.hits.total}}, "iAnalyst" : "{{ctx.payload.aggregate.analyst_name.hits.hits.0._source.iAnalyst}}", "iAnalystHost" : "{{ctx.payload.aggregate.analyst_name.hits.hits.0._source.host}}", "last_error_message" : "{{ctx.payload.aggregate.analyst_name.hits.hits.0._source.message}}", "error_datetime" : "{{ctx.payload.aggregate.analyst_name.hits.hits.0._source.@timestamp}}" }"

       }
  }

}}

Hey,

please take your time to properly format your messages in markdown. This makes it real hard to read.

You can loop through an list of items like this

{{#ctx.payload.aggregate.analyst_name.hits.hits}}
Host: {{_source.host}} Message: {{_source.message}}
{{/ctx.payload.aggregate.analyst_name.hits.hits.}}

--Alex

Thanks Alexander Reelsen. Sorry about formatting. Are there tags to wrap the code.

I tried the recommendations as below. Not sure what I am missing, but I dont see any results

"actions" : {
"log" : {
"logging" : {
"text" : "Encountered: {{ctx.payload.hits.total}} in the last hour on {{#ctx.payload.aggregations.analyst_name.hits.hits}}{{_source.host}}: {{_source.message}}, {{/ctx.payload.aggregations.analyst_name.hits.hits}}"

       }
  }

}

Below is the ouput of the search query I sent in my original post. I am trying to parse results from aggregations tags.

{
"took": 17,
"timed_out": false,
"_shards": {
},
"hits": {
"total": 67,
"max_score": 0,
"hits": []
},
"aggregations": {
"analyst_name": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "mockb-1",
"doc_count": 66,
"group_docs": {
"hits": {
"total": 66,
"max_score": null,
"hits": [
{
"_index": "iad-2017.02.08",
"_type": "iad",
"_id": "AVob5Fd0epr-8jzLv2J3",
"_score": null,
"_source": {
"method": "?",
"thread": "analystDesktop-5637-MacBook-Pro.local6dd6-1",
"message": "I am testing IAD error logging to logstash",
"priority": "ERROR",
"type": "iad",
"path": "net.interactions.ianalyst.jms.IADMessageReceiver",
"@timestamp": "2017-02-08T04:03:53.833Z",
"file": "?:?",
"iAnalyst": "mockb-1",
"@version": "1",
"host": "172.18.187.85:57938",
"logger_name": "net.interactions.ianalyst.jms.IADMessageReceiver",
"class": "?",
"timestamp": 1486526633765
},
"sort": [
1486526633833
]
}
]
}
}
},
{
"key": "mockb-3",
"doc_count": 1,
"group_docs": {
"hits": {
"total": 1,
"max_score": null,
"hits": [
{
"_index": "iad-2017.02.07",
"_type": "iad",
"_id": "AVol6L7Repr-8jzL094Y",
"_score": null,
"_source": {
"message": "I am testing IAD error logging to logstash",
"priority": "ERROR",
"type": "iad",
"@timestamp": "2017-02-08T21:03:53.833Z",
"iAnalyst": "mockb-3",
"host": "172.18.187.85:57938"
},
"sort": [
1486587833833
]
}
]
}
}
}
]
}
}
}

You can use common markdown to format your posts.

Also, please take your time and compare your script example with the path of the json data, you will find the buckets field missing for example.

Thanks Alexander. That was useful.
Future reference on how to iterate the nested arrays in watcher using mustang

"{ "total_errors" : {{ctx.payload.hits.total}},{{#ctx.payload.aggregations.analyst_name.buckets}}{{#group_docs.hits.hits}} {"iAnalyst" : "{{_source.iAnalyst}}", "iAnalystHost" : "{{_source.host}}", "last_error_message" : "{{_source.message}}", "error_datetime" : "{{_source.@timestamp}}"}, {{/group_docs.hits.hits}}{{/ctx.payload.aggregations.analyst_name.buckets}}"

Thanks, this was really helpful for me too. Is there a way to show all hits in the alert rather than just 10?

please open new threads instead of appending to older ones.

You may want to check out size parameter of the terms aggregation

--Alex

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.