Which Rule Type is Better to Monitor the data streams and raise an alert

Elastic Observability
1

Hi Team ,

we are exploring elastic observability. At the moment , using logstash pipelines as intermediary , we are able to push our logs from different applications to central elastic. Every application logs go into separate data stream ( For example : logs-springboot-dev , logs-nodejs-dev ).

Now , we would like to have queries over these data streams and raise an alert if some error is there in logs ( For example : business errors , critical errors with some codes ). Basically to search for specific strings in each data stream indexes and raise an alert.

To do so , I can see - I need to create rules under observability and use either log threshold rule type/elastic query. But under log threshold rule type , I don't see option to specify index/data stream in the query and also can not specify multiple search strings matches pattern in the query.

So little confused which option under observability that suits my requirement to monitor the incoming logs and raise an alert if there is an error in logs.

Please suggest

2

Hi @vinay.bommarati ,

The log threshold rule uses the indices configured in the setting of the Logs app, as shown below: (documentation link)

image
image2876×1320 258 KB

So you can add your indices there and then use log threshold to alert based on that.
Hope that helps

3

Thanks @maryam-saeidi for the reply.

I was looking to filter within the query and able to do that with dataset as a filed as below.
image

Now , I am looking to maintain these ERROR codes highlighted in MATCHES condition in a variable so instead of hardcoding , we can update the variable for new codes. Is it possible and also other than Log Threshold , is there any other way where we can monitor the logs for specific error codes. Just to ensure I am not missing any feature to explore before finalizing on this :slight_smile:

4

We don't have a concept of variables in our alerting rules.

In case you want to be able to change an error message without a need to change the rule definition, one idea can be to use a shared field between different error logs (for example, log.level) to filter all the related documents and then use group by on the message field to get one alert per message. (documentation link)

5

Hi @maryam-saeidi - Thanks for the response.
What my requirement is :

  • Every Application that is sending logs to logstash do have a specific error codes to be monitored for and trigger an email whenever that matches in the incoming message.

I am wondering if indeed "Log Threshold Rule" type is the preferred one for this kind of requirement or Elastic Search Query also can work. I can see there are multiple Rule Types under stack management.

Log Threshold Rule Observations :

  1. If I use "matches phrase" , then it is an exact match. If I use this and my application has many error codes to be monitored , I end up adding more rules i.e. one rule for one error code.
  2. If I use "matches" instead , then it is matching one or more words in the message. Then , when alert generated , we don't know alert got generated for which error code.

So in short , it is not fully satisfying the requirement.

I am also thinking to look at exploring "Elastic Search Rule Type" . But to me , this looks like standard requirement of observability application to be able to look for specific error codes and raise an alert if it matches.