# Log threshold alerting rule to check the presence of logs on specific hosts **URL:** \ **Category:** Kibana\ **Tags:** elastic-stack-alerting\ **Created:** [September 12, 2023, 10:07am UTC](https://discuss.elastic.co/t/log-threshold-alerting-rule-to-check-the-presence-of-logs-on-specific-hosts/342799 "2023-09-12T10:07:53Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![melkamar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/melkamar/32/100785_2.png) [@melkamar](https://discuss.elastic.co/u/melkamar)\ **Post date:** [September 12, 2023, 10:07am UTC](https://discuss.elastic.co/t/log-threshold-alerting-rule-to-check-the-presence-of-logs-on-specific-hosts/342799/1 "2023-09-12T10:07:53Z")
Hi, I am trying to set up an alert rule that will alert me when a job that I expect to run at particular servers stops writing into the syslog. The idea is that I want to receive alerts when: - Any host with a field `server_type: "gitlab-runner"` - Does not have a log message containing the phrase `Total reclaimed space` - In the last 8 hours I set up a Log threshold rule with the configuration ```auto WHEN THE count OF LOG ENTRIES WITH message MATCHES PHRASE "Total reclaimed space" AND server_type IS gitlab-runner IS less than 1 FOR THE LAST 8 hours GROUP BY host.name (check every 2 hours) ``` The problem with this alert rule is that it considers _all_ the incoming hosts. Most of the hosts do not have that `Total reclaimed space` in their logs (and they are not expected to), so they send alerts. That is not what I want. I want something to tell Kibana to "before doing anything else, ignore hosts not annotated with `server_type: gitlab-runner`". How can I do that? Thank you ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [September 12, 2023, 5:46pm UTC](https://discuss.elastic.co/t/log-threshold-alerting-rule-to-check-the-presence-of-logs-on-specific-hosts/342799/2 "2023-09-12T17:46:47Z")
What version are you using? Hi @melkamar What is the mapping type of `server_type` should be `keyword` So a couple of things.... Doing `Less Than` and grouping can have some performance implications that is a longer discussion. Try this I think then you will only get from the hosts that have `server_type` `GROUP BY server_type, host.name` ---
**Author:** ![melkamar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/melkamar/32/100785_2.png) [@melkamar](https://discuss.elastic.co/u/melkamar)\ **Post date:** [September 13, 2023, 10:12am UTC](https://discuss.elastic.co/t/log-threshold-alerting-rule-to-check-the-presence-of-logs-on-specific-hosts/342799/3 "2023-09-13T10:12:02Z")
Hi, I am using 8.7.1. `server_type` is a `keyword`. I tried doing the `GROUP BY` you suggest, but the problem remains. The problem is that, when you say > you will only get from the hosts that have server\_type All my hosts have `server_type` defined. So this will still match everything. What I want is to only match hosts that have a particular value of `server_type`. * * * I saw the warning about not using `Less than` in a grouping query, but I could not think about an alternative. I am very much open to other approaches to this. All I need is an alerting rule that will check that a particular string appears in the logs on a regular basis. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [September 13, 2023, 2:56pm UTC](https://discuss.elastic.co/t/log-threshold-alerting-rule-to-check-the-presence-of-logs-on-specific-hosts/342799/4 "2023-09-13T14:56:22Z")
Yes I under stand So here is my test and it works and only limits to the the ` kubernetes.labels.app IS productcatalogservice` BUT this is a `more than` case ```auto LOG VIEW Default WHEN THE count OF LOG ENTRIES WITH message MATCHES via_upstream AND kubernetes.labels.app IS productcatalogservice Add condition IS more than 2000 FOR THE LAST 5 minutes GROUP BY kubernetes.labels.app, host.name ``` ![Screenshot 2023-09-13 at 7.13.12 AM](https://us1.discourse-cdn.com/elastic/original/3X/6/6/66c008b3c7d2330e853172169ce018b5aaaa8e81.png) BUT when I tried it with a `less than` case I got all the `kubernetes.labels.app` so you are right I think that an artifact of the "Less Than" ... and more I look at it I get why ... ```auto LOG VIEW Default WHEN THE count OF LOG ENTRIES WITH message MATCHES via_upstream AND kubernetes.labels.app IS productcatalogservice Add condition IS less than 10000 FOR THE LAST 5 minutes GROUP BY kubernetes.labels.app, host.name ``` Because in fact the condition IS actually met for every pod and host because there are 0 entries for all the conditions/combination so that is why all are being reported... with the More Than those conditions are not Met. So all that ... hmmm yup .... I need to think about that so in the End if you are really just trying to figure out when the last time a particular service wrote a log... I would use perhaps a latest transform and then a simple alert on top of that > **[Transform overview | Elasticsearch Guide \[8.10\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-overview.html#latest-transform-overview)** Latest transform are pretty awesome way of keeping track of the "Last Event" from logs, services, hosts etc.... give a look. I have used this more many use cases.... it works really well! There also may be another way to do this with a DSL Query. ---
**Author:** ![Andrew\_Mora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_mora/32/125622_2.png) [@Andrew\_Mora](https://discuss.elastic.co/u/Andrew_Mora)\ **Post date:** [September 13, 2023, 4:35pm UTC](https://discuss.elastic.co/t/log-threshold-alerting-rule-to-check-the-presence-of-logs-on-specific-hosts/342799/5 "2023-09-13T16:35:46Z")
> [@melkamar](#): > > Hi, > > I am trying to set up an alert rule that will alert me when a job that I expect to run at particular servers stops writing into the syslog. The idea is that I want to receive alerts when: > > - Any host with a field `server_type: "gitlab-runner"` > - Does not have a log message containing the phrase `Total reclaimed space` > - In the last 8 hours > > I set up a Log threshold rule with the configuration > > ```auto > WHEN THE count OF LOG ENTRIES > WITH message MATCHES PHRASE "Total reclaimed space" > AND server_type IS gitlab-runner > > IS less than 1 > FOR THE LAST 8 hours > GROUP BY host.name > > (check every 2 hours) > > ``` > > The problem with this alert rule is that it considers _all_ the incoming hosts. Most of the hosts do not have that `Total reclaimed space` in their logs (and they are not expected to), so they send alerts. That is not what I want. I want something to tell Kibana to "before doing anything else, ignore hosts not annotated with `server_type: gitlab-runner`". How can I do that? > > Thank you In your case, the filter clause would be: `filter: - host.name: type: equals value: gitlab-runner` This filter clause tells Kibana to only consider hosts that have the `server_type` field set to `gitlab-runner`. So, the complete alert rule would be: `WHEN THE count OF LOG ENTRIES WITH message MATCHES PHRASE "Total reclaimed space" AND filter: - host.name: type: equals value: gitlab-runner IS less than 1 FOR THE LAST 8 hours GROUP BY host.name (check every 2 hours)` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [September 13, 2023, 4:45pm UTC](https://discuss.elastic.co/t/log-threshold-alerting-rule-to-check-the-presence-of-logs-on-specific-hosts/342799/6 "2023-09-13T16:45:38Z")
Hi @Andrew_Mora Welcome to the community and thanks for the help. Can you show a screenshot where you see a `filter` on the Log Threshold Rules Screen? Perhaps I am missing it somewhere ... On other rules there is a KQL filter but Log Threshold does not have it Can you show us on this screen where you see the filter option is? This is version 8.9.1 so pretty up to date what version do you see filter on? ![Screenshot 2023-09-13 at 9.42.12 AM](https://us1.discourse-cdn.com/elastic/original/3X/1/7/17a84feed1c421e2740a380e4901793511be8780.png) ---
**Author:** ![melkamar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/melkamar/32/100785_2.png) [@melkamar](https://discuss.elastic.co/u/melkamar)\ **Post date:** [September 19, 2023, 9:40am UTC](https://discuss.elastic.co/t/log-threshold-alerting-rule-to-check-the-presence-of-logs-on-specific-hosts/342799/7 "2023-09-19T09:40:28Z")
Yeah I ran into the same issue and also conclusion - it is _technically_ correct that all the hosts are matched, because the condition applies to all 😁 Huge thank you for pointing me in the direction of the Transform feature though. I have not used it before and it's exactly what I need! That solves my initial question. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 17, 2023, 9:41am UTC](https://discuss.elastic.co/t/log-threshold-alerting-rule-to-check-the-presence-of-logs-on-specific-hosts/342799/8 "2023-10-17T09:41:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.