Observability logs - Alert log stoppage

[jancodenew (jan)]

Hello,

I am able to create log stoppage alert using threshold alert type in logs-ovservability.

Is there any way to add multiple index names in this observalibilty-logs settings?, so that I can create multiple alerts for log stoppage on each servers/devices in specific index.

Now I am able to add only one index.

[weltenwort (Felix Stürmer)]

Hi @jancodenew,

currently the log threshold alerts are coupled to the Logs UI configuration in the active space. So within the same space there there unfortunately is no easy way to achieve that. As a workaround you could create additional spaces, configure the Logs UI to target different indices and create alerts in them.

Alternatively, maybe the more generic query threshold alert available from the "Stack Management" / "Rules and Connectors" UI can give you more flexibility within one space?

[jancodenew (jan)]

Thank you @weltenwort for your reply.

Sorry, I didn't understand the workaround mentioned by you. what do you meant by create additional spaces, configure the Logs UI to target different indices and create alerts in them?

[weltenwort (Felix Stürmer)]

Kibana has the concept of "spaces", which are separate workspaces for Kibana entities like visualizations and dashboards. You can manage them via the corresponding Stack Management page:

The alerts use the Logs UI settings from the space they are created in. So if you configure the Logs UI to target different indices in different spaces, the alerts created in the respective spaces will also target these indices:

The alternative I mentioned was to use an "Elasticsearch query" alert with a "below 1" threshold:

Hope that helps a bit

[jancodenew (jan)]

@weltenwort Yea that really helps a lot.
I have one more query for a clarification. I have more than 60 indices in the current ELK SIEM setup. Is it ok to create 60 spaces to handle this ?
Will that make a impact to the platform?

[weltenwort (Felix Stürmer)]

Not 100% sure, but intuitively I wouldn't expect that. A space is just a "passive" concept to partition Kibana entities so there shouldn't be a difference between 60 alerts in one space or one alert in 60 spaces each.

[jancodenew (jan)]

Hi @weltenwort

I have tried creating the log stoppage alert with the second option which you have mentioned("Elasticsearch query" alert with a "below 1" threshold).

Could you please share a sample query here to group by host.hostname in Elasticsearch query?

Log stoppage alert Rule Logic: Log stoppage for last 15min from each hostname should be triggered.

Thanks in advance

[weltenwort (Felix Stürmer)]

That is not possible with the "Elasticsearch query". In general, alerting for missing log data on dynamic groups is problematic. How would the alert know that a host exists if it is not sending data?

The Logs Threshold performs some dirty tricks to work around that by looking at a larger time range to determine the groups, but then evaluate the conditions for a smaller time window. This is at the cost of performance, of course.