Custom variables in Kibana Alerts

Elastic Stack Kibana
1

I am exploring alerts and connectors in Kibana. I can configure and test them perfectly but I am unable to use the custom variable present in metricbeat, filebeat or any other beat module index. Let me explain this by an example.

Let say, two different snapshot of my application is running on different servers with metricbeat docker module enabled. The hostname of my first server is host1 and second is host2. The container name of the application is myapp. I've one variable serviceName which is a combination of hostname and conatiner name (host1_myapp, host2_myapp).
I can get either the docker name or hostname by mentioning them to the context group but not any other variable like serviceName.

Is there any way to use the custom variable in the Kibana alerts?

2

If I understand the issue correctly that you are trying to get the combined values of hostname and container name in a field within context group, I think using a scripted field might work in this situation. You can create a new scripted field that is generated from the combined value of hostname and container name and you can reference that field in the context group to get the value you are looking for.

3

@Hung_Nguyen, thanks for your reply. I tried scripted fields as well but it doesn't work. Kibana unable to access the scripted field at the time of the alert. Could you please be more specific and tell me some example or articles where a similar use-case listed?

5

Hey Ankur,

scripted fields don't seem to be accessible from watchers or alerts as it is created on the fly in Kibana so my bad. Yeah I am not really sure how we can do this in Kibana Alerts at the moment. I think it might worth your while to look into querying the results from your detection/rule in the .siem-signals* index using a watcher. It should give you more flexibility

6

Hi @Ankur_Mahajan,

What type of alert / trigger type are you trying to create (ex: log threshold)? or is this alert you're creating from the alerting management UI or from a specific application? (APM, Uptime, etc)

There's been similar requests on some types of alerts and I can findout if it is currently possible or if there is an enhancement request opened based on what type of alert you're using.

7

@mikecote thanks for your reply, I am trying multiple alerts type like log threshold, inventory and metric threshold. I want to access some fields which are present in my filebeat or metricbeat index like instanceName but no luck so far. Is there any way to access some custom fields in alerts?

8

Perhaps if you try the Create Alert per Setting and set it to ServiceName you can get what you are looking for.

I was just describing this in another thread

9

@stephenb, thanks for your reply. I checked the other ticket as well and he is also looking for the same thing. I want to access some custom fields let say application name which is there in the index but I am unable to get in the alert context. I am not asking this specifically to hostname or container name that I can get by context but not both at the same time. According to your suggestion if I create an alert for a service that would lead to disaster (assume I've 1000 docker containers) as I've to maintain multiple alerts and indexes. I want to do this in a more generic way means one alert for a type and I can manage multiple application in that by some conditional fields would be an efficient way to do this.

Any help would be highly appreciated.

10

This probably won't help but perhaps it .... let me know if I am on track.

An alert is really when an aggregation crosses a threshold.

Avg CPU over 5 min > 80%

That could be made up of 1 doc / sample or 1000 docs / sample

That aggregation is across some dimensions host, container service etc... And let's just say for info sake that is the hierarchy

Many Hosts each with many Containers each with many services.

And I have also added fields / keywords to the beats collecting those metrics.

The issue is unless you have filtered or grouped by the field you want to report on the aggregation could have a number of different values possible for those fields you added.

So back to

Host AVG CPU > 80% for 5 min

Could have many different containers and it services that contribute so when you want to add that field / value to the alert which one would it be? It could have different values.

If we are saying that after you aggregate you will have 5 fields that exist in every doc with the same value that made that aggregation up ... Yes I think that is a reasonable request perhaps to put in an alert.... But I am not sure...

Am I thinking of this correctly?

11

Yes @stephenb , you are on track but let me explain it once again. The field that I am talking about could have different values based on the services/containers/host. Actually, I want to create a mechanism where I can filter out the alerts based on these fields. Let me give you an example of the log threshold. Let say I've filebeats running on multiple servers and the payload that I receive from them are below.

[
  {
    "log": {
      "level": "error"
    },
    "host": "server1",
    "message": "A sample message.",
    "containers": {
      "name": "container1"
    },
    "customField": "customValue1"
  },
  {
    "log": {
      "level": "info"
    },
    "host": "server2",
    "message": "A sample message.",
    "containers": {
      "name": "container1"
    },
    "customField": "customValue2"
  },
  {
    "log": {
      "level": "error"
    },
    "host": "server3",
    "message": "A sample message.",
    "containers": {
      "name": "container2"
    },
    "customField": "customValue3"
  },
  {
    "log": {
      "level": "error"
    },
    "host": "server1",
    "message": "A sample message.",
    "containers": {
      "name": "container1"
    },
    "customField": "customValue1"
  }
]

Alert body

{
  "alert_id": "{{alertId}}",
  "alert_name": "{{alertName}}",
  "space_id": "{{spaceId}}",
  "tags": "{{tags}}",
  "alertInstanceId": "{{alertInstanceId}}",
  "context_matchingDocuments": "{{context.matchingDocuments}}",
  "context_conditions": "{{context.conditions}}",
  "myCustomField": "{{customField}}"
}

I want to access "myCustomField": "{{customField}}" and build a conditional framework on top of this but currently I am unable to do so.

12

Thanks for the explanation ... Understood, shall we proceed?

Now lets follow this through with your example

So now as sample alert could be

WHEN log.level is error GREATER THAN 3 times in 1 minute

And that triggers the alert...

The alert is an aggregation... so there is more than 1 doc that was aggregated.

Which value of customField do you expect to be in the alert body? Since there are 4 docs with 3 different values of customField ... "customValue1", "customValue2" or "customValue3"

13

@stephenb, I want the variable for which the alert is triggered.

It would be better if we not take the example of the log threshold. The actual use-case I am looking is the inventory.

[
  {
    "@timestamp": "2020-09-24T17:03:33.790Z",
    "host": {
      "name": "server1"
    },
    "customField": "customValue1",
    "event": {
      "duration": 143911,
      "dataset": "system.cpu",
      "module": "system"
    },
    "metricset": {
      "name": "cpu",
      "period": 10000
    },
    "service": {
      "type": "system"
    }
  },
  {
    "@timestamp": "2020-09-24T17:03:33.790Z",
    "host": {
      "name": "server2"
    },
    "customField": "customValue2",
    "event": {
      "duration": 143911,
      "dataset": "system.cpu",
      "module": "system"
    },
    "metricset": {
      "name": "cpu",
      "period": 10000
    },
    "service": {
      "type": "system"
    }
  },
  {
    "@timestamp": "2020-09-24T17:03:33.790Z",
    "host": {
      "name": "server3"
    },
    "customField": "customValue3",
    "event": {
      "duration": 143911,
      "dataset": "system.cpu",
      "module": "system"
    },
    "metricset": {
      "name": "cpu",
      "period": 10000
    },
    "service": {
      "type": "system"
    }
  },
  {
    "@timestamp": "2020-09-24T17:03:33.790Z",
    "host": {
      "name": "server1"
    },
    "customField": "customValue1",
    "event": {
      "duration": 143911,
      "dataset": "system.cpu",
      "module": "system"
    },
    "metricset": {
      "name": "cpu",
      "period": 10000
    },
    "service": {
      "type": "system"
    }
  },
  {
    "@timestamp": "2020-09-24T17:03:33.790Z",
    "host": {
      "name": "server1"
    },
    "customField": "customValue1",
    "event": {
      "duration": 143911,
      "dataset": "system.cpu",
      "module": "system"
    },
    "metricset": {
      "name": "cpu",
      "period": 10000
    },
    "service": {
      "type": "system"
    }
  },
  {
    "@timestamp": "2020-09-24T17:03:33.790Z",
    "host": {
      "name": "server1"
    },
    "customField": "customValue1",
    "event": {
      "duration": 143911,
      "dataset": "system.cpu",
      "module": "system"
    },
    "metricset": {
      "name": "cpu",
      "period": 10000
    },
    "service": {
      "type": "system"
    }
  }
]

This is a sample metric-beat JSON with limited information. Let's assume server1 and server3 are reaching the threshold for CPU utilization. So when the alert is triggered for both of these events I want to get customField values for corresponding servers (server1 = customValue1 and server3 = customValue3).

14

Interesting... Both of those would be enhancements for sure... Not even sure how those we would be handled...Max would be a sub aggregation of the docs that made up the alert ... A list could be very large ...

As @mikecote suggested perhaps open an enhancement required.

15

@stephenb please check the updated comment.

16

That is still not supported today as @mikecote stated, even if we assume that customField's value is the same for every server instance...

So perhaps fill out an enhancement request in the Kibana GitHub repo.

You could get the value of custom field if you created alert by on that custom field, understand you don't want to do that but that is a workaround which I've implemented for another customer. it doesn't allow you to get three or four custom Fields though but you could get one.

1 Like
17

@stephenb, thank you for your support and time. I really appreciate your help.

I'll create an enhancement request in the kibana github repository.

1 Like
18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.