Alerting on log data

[DanRoscigno]

Here is some information that is in my logs:

I would like to be able to create alerts any time a log entry includes the word ERROR, and the kubernetes.pod.name includes the string redis-leader. I have removed the match against ERROR in the log as this works fine, I am only having a problem matching on redis-leader within kubernetes.pod.name. I am using the log threshold alert type. Here are some things I tried:

This does not work:

WHEN more than or equals 1 log entry 
WITH kubernetes.pod.name IS redis-leader-*
FOR THE LAST 5 minutes

This works, but I have to add an alert for every pod, and if the app scales the new pods are unmonitored because the pod gets a fresh name:

WHEN more than or equals 1 log entry 
WITH kubernetes.pod.name IS redis-leader-74d59c4b7f-zxz81
FOR THE LAST 5 minutes

Should I be able to use wildcards? I tried a bunch of escaping and quoting, but never got there.

The next thing I would like to do is to add something like a group by, as an alert that tells me that there were five error messages for redis-leaders is less valuable that 3 separate alerts telling me that redis-leaders had messages containing errors. I cannot find a group by in the UI for log thresholds. It would be great if I could use the technique from Create alert per multiple fields

[weltenwort (Felix Stürmer)]

Hi @DanRoscigno,

the ability to create grouped alerts will be included in the upcoming 7.9.0 release:

Wildcards are currently not supported in the field comparison operators. On analyzed fields the matches operator is available, which is uses the match Elasticsearch clause. Other operators will probably be added in the future.

For your use-case it sounds like you would be best served by using an is operator to filter by some keyword field that exactly identifies all redis leaders (e.g. the image name? or a tag/label?) in combination with a group by kubernetes.pod.name.

[DanRoscigno]

Thanks @weltenwort ! I spun up a 7.9 cluster and the group by is perfect. I definitely see a need for searching within keyword fields, and the upcoming wildcard fields. Should I put in an ER or is that already on the list?

[weltenwort (Felix Stürmer)]

Glad it seems useful to you! I couldn't find anything that tracks the addition of such an operator on keyword fields, so an ER would definitely increase the chance that it'll be added. Thank you!