Create and alert based on a GROUP BY

Elastic Observability Logs
1

Hi Community,

I have a system that logs a JSON object containing a user_id field into a document, and I need to create an alert that notifies me if a user makes more than a certain number of accesses within a certain time frame. I tried creating a log threshold alert, but I don't have access to the fields of the message. Alternatively, if I try to create an alert with KQL query, I don't have access to the group by function. Any advice on how to solve this?

1 Like
2

Hi Matteo, Welcome to the Elastic communtiy.

  1. May I know what kind of access you have currently ?
  2. You can refer log threshold alert to set the same but it required count and group by command.
3

Hi,
thanks for your quick response.
I am currently an "organization owner" on the ELK deployment. I red those docs this morning, still, I didn't understand how to access the fields of the json object (e.g. how to do message.used_id IS sth)

4

You can access nested field using dot(.) only. Could you please share your sample json document? and on which field you want to perform count ?

5

Sure,
The structure of the "message" is:
{
"AuthenticationSuccess": {
"status": 200,
"user_id": "test_user",
"message": "OK",
"source": "portal"
}
}
What I need to do is acces the user_id field from the following alert section

image
image670×619 42.2 KB

but even using dot(.) I cannot access it.
What I am trying to achieve is raise an alert if a user tries to login too many times in a certain time frame (so I am trying to group by user and then count and raise the alert)

6

This is an example of the log in the "Discovery" section

image
image963×102 8.6 KB

7

As far I understand, You should access like this AuthenticationSuccess.user_id. Why you trying to access from message field?

I am assuming you have shared json doc structure.

8

Yes, my bad.
Still, it doesn't find the field
image
I tried with "message.AuthenticationSuccess.user_id" as well

9

Quick update: I did manage to solve my issue by using the "Alerts" section under "Security" instead of using the one in "Stack management"

10

I'm running into the same issue - many fields are missing from the Grouped Over menu options in Alerts.
The one in particular that I would like to use is cluster_name. It is not nested, and is mapped statically in the index template. Every event has this field in it, and yet
image
image

As I do not have the appropriate license to use the Security alert feature, I can't use that solution.
Any idea why are some fields not available to group by...?

Full JSON

{
"_index": "vcmts-beats-nocontainer-2024.09.11",
"_id": "tiLD4pEBhUFuJomPPsZu",
"_version": 1,
"_score": 0,
"_source": {
"tags": [
"beats_input_codec_plain_applied"
],
"@timestamp": "2024-09-11T20:25:31.610Z",
"message": "Trace[1770768237]: ---"Listing from storage done" 805ms (20:25:30.825)",
"cluster_name": "ClusterName",
"type": "vcmts",
"node_name": "",
"event": {
"original": "Trace[1770768237]: ---"Listing from storage done" 805ms (20:25:30.825)"
},
"@version": "1",
"app": "platform.kube_apiserver",
"site_name": ""
},
"fields": {
"app": [
"platform.kube_apiserver"
],
"cluster_name": [
"ClusterName"
],
"event.original": [
"Trace[1770768237]: ---"Listing from storage done" 805ms (20:25:30.825)"
],
"node_name": [
""
],
"tags.keyword": [
"beats_input_codec_plain_applied"
],
"@version.keyword": [
"1"
],
"message": [
"Trace[1770768237]: ---"Listing from storage done" 805ms (20:25:30.825)"
],
"type": [
"vcmts"
],
"tags": [
"beats_input_codec_plain_applied"
],
"site_name": [
""
],
"@timestamp": [
"2024-09-11T20:25:31.610Z"
],
"@version": [
"1"
],
"app.keyword": [
"platform.kube_apiserver"
]
}
}