I'm trying to set up a Watcher in Kibana that will let me find an log with the message "Exit Status 1", then use the timestamp from that log to chain a new search and go back 20 logs or so from there so I can kind of create a stacktrace in Slack so we can begin to think about what might have caused the error without actually firing sorting through the logs.
Here's what I have so far:
{
"trigger": {
"schedule": {
"interval": "5m"
}
},
"input": {
"chain": {
"inputs": [
{
"first": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"pcf-*"
],
"types": [],
"body": {
"size": 1,
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "cf_app_name:\"app-name\""
}
},
{
"query_string": {
"query": "msg:\"*Exit status 1*\""
}
},
{
"query_string": {
"query": "cf_space_name: prod"
}
}
]
}
}
}
}
}
}
},
{
"second": {
"transform": {
"script": {
"source": "return ['time': ctx.payload.first.hits.hits.0._source.syslog5424_ts = new date_time()]",
"lang": "painless"
}
}
}
},
{
"third": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"pcf-*"
],
"types": [],
"body": {
"size": 20,
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "cf_app_name:\"app-name\""
}
},
{
"query_string": {
"query": "cf_space_name: prod"
}
},
{
"range": {
"time": {
"gte": "ctx.payload.second.time - 5m"
}
}
}
]
}
}
}
}
}
}
}
]
}
},
"condition": {
"always": {}
},
"actions": {
"notify_slack": {
"slack": {
"message": {
"to": [
"#my-team"
],
"text": "{{ctx.payload.second.time}}{{ctx.payload.third.hits.hits.0._source.syslog5424_ts}} {{ctx.payload.third.hits.hits.0._source.msg}} \n {{ctx.payload.third.hits.hits.1._source.msg}} \n {{ctx.payload.third.hits.hits.2._source.msg}} \n {{ctx.payload.third.hits.hits.3._source.msg}} \n {{ctx.payload.third.hits.hits.4._source.msg}} \n {{ctx.payload.third.hits.hits.5._source.msg}} \n {{ctx.payload.third.hits.hits.6._source.msg}} \n {{ctx.payload.third.hits.hits.7._source.msg}} \n {{ctx.payload.third.hits.hits.8._source.msg}} \n {{ctx.payload.third.hits.hits.9._source.msg}} \n{{ctx.payload.third.hits.hits.10._source.msg}} \n {{ctx.payload.third.hits.hits.11._source.msg}} \n {{ctx.payload.third.hits.hits.12._source.msg}} \n {{ctx.payload.third.hits.hits.13._source.msg}} \n {{ctx.payload.third.hits.hits.14._source.msg}} \n {{ctx.payload.third.hits.hits.15._source.msg}} \n {{ctx.payload.third.hits.hits.16._source.msg}} \n {{ctx.payload.third.hits.hits.17._source.msg}} \n {{ctx.payload.third.hits.hits.18._source.msg}} \n {{ctx.payload.third.hits.hits.19._source.msg}}"
}
}
}
}
}
Based on the different things I've tried, I either get an error saying that the time range I'm trying to specify can't be parsed because it's not an instance of a date/time or I get zero results and I don't know why that is