Return a few log lines before and after an ELASTIC watcher match query

[mark-otcxn]

I have an ELASTIC watcher that searches for "ERROR" in logs, and then sends an e-mail when this happens. Is it possible for the watcher to return, say the next 2 or 3 lines after the ERROR. This could help give additional context to the ERROR. Is it also possible to include 2 or 3 lines before the ERRROR

{
"trigger": {
"schedule": {
"interval": "1m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"test_log_qa*",
],
"types": [],
"body": {
"query": {
"bool": {
"must": [
{
"match_phrase": {
"message": "ERROR"
}
},
{
"range": {
"@timestamp": {
"gte": "now-1m"
}
}
}
]
}
},
"sort": [
{
"@timestamp": {
"order": "desc"
}
}
]
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 0
}
}
},
"actions": {
"send_email": {
"email": {
"profile": "gmail",
"to": [
"test@test.com"
],
"subject": "QA ERROR found in logs",
"body": {
"text": "ERROR found in logs.\n\n Details: \n\n\n\n{{#ctx.payload.hits.hits}}Host: {{_source.host}}\nComponent:{{_source.path}}\n{{_source.message}}\n\n{{/ctx.payload.hits.hits}}"
}
}
}
}
}

[spinscale (Alexander Reelsen)]

I moved this over to the x-pack forum. Also, please take some time to properly format you code snippets, as you can just use markdown in this forum.

The hard part of this question is what does after the error mean. If you are aggregating logs, then the next two messages based on time could be from two completely different systems and thus being fully useless. If you filter on the same host, maybe the next two lines are from a different service and thus not usable as well. You need to define for yourself what next means and then come up with a proper query for this.

So, here is what you could do:

Each action, can execute a so called search transform, using this you can execute a second search query, that executes roughly the same search but maybe filters for the hostname or service in addition plus a time filter that begins exactly where the hit occured. Note that this only makes sense if you only have one hit as a response. If you have 10 different hosts, then the query will be tough to write.

Hope this helps.

[mark-otcxn]

@spinscale

e filters for the hostname or service in ad

Hi Alex, thanks for your reply. 'next' would mean the next few lines in that same log file. The search results we have so far can come from multiple hosts, so what we would be looking to do would be to report the 'next few' lines in each case, rather that just a single line. Is search transform still the approach you think?

[spinscale (Alexander Reelsen)]

hey,

yes, you can try with a search transform. The main question here is not so much about watcher, but more about the ability to write a query that answers your question and context of a a few lines before and after... so I'd test out if I am able to write a proper query, and then go from there.

--Alex