Kibana watcher returns correct data for ID but wrong data for name

Hello there,

We have a bunch of docker containers running on our application servers and each container is dedicated to a specific service of the application.

We have also setup Kibana monitoring for all these services.

Each of these containers has a container ID which is being logged in the Kibana payload under the field HOSTNAME
And the same service is also represented with the help of another identifier which is the name of the service (for ex: login-app-service). This service name is logged in the Kibana payload under the field app.name and also under another field called application

I am writing a watcher script to retrieve all the ERRORS occurring in the login-service once every hour. However, if I try to identify the logs based on the name of the service i.e., as shown below, then the watcher returns false data that belongs to a completely different service of the application (ex: navigate-service).

{
"match":
{
"app.name": "login-service"
}
}

This is returning all the errors being logged under navigate-service instead of login-service

But if I use the container ID as the identifier as follows, I get the correct output:

{
"match":
{
"HOSTNAME": "abcde123fgh2"
}
}

However, this will introduce a dependency of having to go back to Kibana and change this filter manually everytime a new image of the container is deployed.

Any idea if I can refactor my watcher scripts such that it can still return the expected output using the app.name itself as the filter?

Screenshot depicting the fields in my payload is attached. Also mentioned below is the watcher script that I have written for this purpose.

{
"trigger": {
"schedule": {
"interval": "1h"
}
},
"input": {
"chain": {
"inputs": [
{
"first": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"-"
],
"types": [],
"body": {
"query": {
"constant_score": {
"filter": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-1h"
}
}
}
],
"must": [
{
"match": {
"level": "ERROR"
}
},
{
"bool": {
"must": [
{
"match": {
"host": ""
}
},
{
"match": {
"HOSTNAME": "abcdef1234gh"
}
}
]
}
}
]
}
}
}
},
"_source": [
"req.requestURI",
"message",
"application",
"@timestamp"
]
}
}
}
}
},
{
"second": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"-
"
],
"types": ,
"body": {
"query": {
"constant_score": {
"filter": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-1h"
}
}
}
],
"must": [
{
"match": {
"level": "ERROR"
}
},
{
"bool": {
"must": [
{
"match": {
"host": ""
}
},
{
"match": {
"HOSTNAME": "dfsrt1234oit"
}
}
]
}
}
]
}
}
}
},
"_source": [
"req.requestURI",
"message",
"application",
"@timestamp"
]
}
}
}
}
}
]
}
},
"condition": {
"compare": {
"ctx.payload.first.hits.total": {
"gt": 0
}
}
},
"actions": {
"send_email": {
"email": {
"profile": "standard",
"to": [
"myemailaddress@domain.com"
],
"subject": "ERRORS in my application - login service over the past 1 hour",
"body": {
"text": "{{ctx.payload.first.hits.total}} errors have occurred in login service of <server 1> in the past 1 hour \n\n The error messages that have been logged over the past 1 hour can be found below:\n\n {{#ctx.payload.first.hits.hits}}{{_source}}:\n{{/ctx.payload.first.hits.hits}}\n\n\n\n{{ctx.payload.second.hits.total}} errors have occurred in login service of <server 2> in the past 1 hour \n\n The error messages that have been logged over the past 1 hour can be found below:\n\n {{#ctx.payload.second.hits.hits}}{{_source}}:\n{{/ctx.payload.second.hits.hits}}"
}
}
},

"throttle_period_in_millis": 3600000
}

try app.name.keyword for an exact match, otherwise a search for login OR service is exeucted.

Hello @spinscale

It worked like a charm! :slight_smile:

Thank you so much! :slight_smile:

The only change I made was replacing the below block as follows:

Block that was present earlier:

{
"match": {
"HOSTNAME": "xxxxxxx"
}
}

Replaced this with:

{
"match": {
"app.name.keyword": "login-service"
}
}

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.