Hello there!
We are currently using Elasticsearch in combination with Kibana and Filebeat to report errors to Slack. This is working perfectly but we would like to show the latest 5 errors in that same Slack message.
This is currently my working watcher (i cant wrap the watcher below into code, weird, the editor does not work):
{
"trigger": {
"schedule": {
"interval": "1m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"log*"
],
"types": ,
"body": {
"size": 0,
"query": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"gte": "now-1m",
"lte": "now"
}
}
},
{
"match": {
"severity": "ERROR"
}
}
]
}
},
"aggs": {
"group_by_domain": {
"terms": {
"field": "domain.keyword",
"size": 1
}
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gte": 1
}
}
},
"actions": {
"notify-slack": {
"throttle_period_in_millis": 300000,
"slack": {
"account": "",
"message": {
"from": "{{#ctx.payload.aggregations.group_by_domain.buckets}}{{key}}{{/ctx.payload.aggregations.group_by_domain.buckets}}",
"to": [
"#channelname",
"@channel"
],
"text": "ERROR - Application Errors.",
"attachments": [
{
"color": "danger",
"title": "Let op!",
"text": "Error occoured for domain {{#ctx.payload.aggregations.group_by_domain.buckets}}{{key}} ({{doc_count}} errors) {{/ctx.payload.aggregations.group_by_domain.buckets}}! Check {{#ctx.payload.aggregations.group_by_domain.buckets}}{{key}}{{/ctx.payload.aggregations.group_by_domain.buckets}} for more information."
}
]
}
}
}
}
}
It shows the amount of errors by domain in the last 1 minute, and it is working prefectly!
But now i want to expand this message by including the last 5 errors in the slack message so the developer can check it without opening kibana first. I have modified the watcher by adding an aggegration:
"group_by_message": {
"terms": {
"field": "message.keyword",
"size": 5
}
}
And adding the group by message to the slack attachement:
\n{{#ctx.payload.aggregations.group_by_message.buckets}}{{key}}\n\n\n{{/ctx.payload.aggregations.group_by_message.buckets}}\n
This does work, sometimes. Sometimes the variables are just empty, and i am unable to find out what is causing this to be empty. If i simulate the watcher, the body is also sometimes filled with the error message.