# Alert on no data per source? **URL:** \ **Category:** Elasticsearch\ **Created:** [March 26, 2025, 3:30pm UTC](https://discuss.elastic.co/t/alert-on-no-data-per-source/376425 "2025-03-26T15:30:52Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Mike8](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike8/32/140538_2.png) [@Mike8](https://discuss.elastic.co/u/Mike8)\ **Post date:** [March 26, 2025, 3:30pm UTC](https://discuss.elastic.co/t/alert-on-no-data-per-source/376425/1 "2025-03-26T15:30:52Z")
[here](https://discuss.elastic.co/t/alert-when-data-is-missing/307886/2) @leandrojmp suggested a Log Threshold to get reported when no logs are added in the last 5 mins. This works but only if you add the specific source as a `WITH` condition. In my case I want to be reported on say 6 sources (but not 2 other stale sources in the index history). It seems it's not suitable in this case at least in the UI since you can only have `WITH...IS...AND` where the more suitable would be something like `WITH source IN [a, b, c]` and `GROUP BY source`. Is there any alternative besides manually creating 1 alert per source? The closest I've gotten is Query DSL with min\_doc\_count 0 which displays the aggs as expected in Dev Tools (e.g. `a: 0, b: 16, c: 44`) but either doesn't work in the alert UI or I'm not setting it right (I tried WHEN sum/count grouped over `source` is below 1) ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 26, 2025, 9:22pm UTC](https://discuss.elastic.co/t/alert-on-no-data-per-source/376425/2 "2025-03-26T21:22:23Z")
Hi @Mike8 What version are you on? And when you say by source... Does that mean source type like nginx logs Or does that source mean by host? Perhaps clarify a bit... On Newer version there is an Alerts -. Observability Alert -\> Custom Threshold Alerts Which will alert per "Group By" including when no data So this says Alert when above X or when there is not data by host.name ![Screenshot 2025-03-26 at 2.25.37 PM](https://us1.discourse-cdn.com/elastic/original/3X/8/c/8c146703744d2fd516c2e83154709f47abd14613.png) ![Screenshot 2025-03-26 at 2.24.13 PM](https://us1.discourse-cdn.com/elastic/original/3X/7/6/76ff575e53d894f449b80ea5ffcfaec1dc34e6ec.png) ![Screenshot 2025-03-26 at 2.30.22 PM](https://us1.discourse-cdn.com/elastic/original/3X/1/f/1f0aa83390046c0710c7eaab80a2dd5289b18d37.png) ---
**Author:** ![Mike8](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike8/32/140538_2.png) [@Mike8](https://discuss.elastic.co/u/Mike8)\ **Post date:** [March 26, 2025, 11:53pm UTC](https://discuss.elastic.co/t/alert-on-no-data-per-source/376425/3 "2025-03-26T23:53:56Z")
Version is 8.17.0. I didn't try Custom type before. Source is basically cluster name. When I left the default, it triggered, but that was triggering for logs being too high. Instead I tried: COUNT all documents EQUATION A IS BELOW 1 FOR THE LAST 5 minutes Group alerts by: source/cluster I also checked "Alert me if there's no data" 1 source has many logs; 1 has no logs for weeks now. But alerts only seem to trigger if I change IS BELOW to IS ABOVE. It seems like many of these alert types don't consider buckets if doc\_count is 0. I tried to limit it by adding a query filter like `source: "live-cluster" OR source: "dead-cluster"` but it didn't help, mainly because the dead-cluster I would ultimately want to exclude from consideration (but for now it's good to attempt to trigger the alert). ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 27, 2025, 1:38am UTC](https://discuss.elastic.co/t/alert-on-no-data-per-source/376425/4 "2025-03-27T01:38:49Z")
One caveat it has to get logs from a source before it can be considered missing. That's the only way it can know if something's missing... If you have 10 sources that are logging and two go stop sending logs those will be alerted on If you add an 11th source that's never logged there's no way to know it's missing data I've tested these cases and they work. I will tell you using them below 1 is not a good approach... For the exact reason that you say when a bucket is zero there's nothing to report on... That's why you want to use the alert me if missing data because that way we're keeping track ---
**Author:** ![Mike8](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike8/32/140538_2.png) [@Mike8](https://discuss.elastic.co/u/Mike8)\ **Post date:** [March 27, 2025, 7:59am UTC](https://discuss.elastic.co/t/alert-on-no-data-per-source/376425/5 "2025-03-27T07:59:22Z")
> [@stephenb](#): > > If you have 10 sources that are logging and two go stop sending logs those will be alerted on When you say logging, you mean based on the alert? Because old logs are still accessible via explorer when filtering for `source: "dead-cluster"`, On the fence between this and multiple `Log Threshold alerts` then; at least with the log threshold I can test it / be confident it works & how the email looks ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 27, 2025, 1:36pm UTC](https://discuss.elastic.co/t/alert-on-no-data-per-source/376425/6 "2025-03-27T13:36:16Z")
Hi @Mike8 Apologies, I should have been more clear If you have three hosts and they are sending telemetry to elasticsearch and in this case I was referring to say application or system logs But it could be anything. It could be metric data etc. And you set up an alert like I showed before that's looking for number of error messages greater than 100 over the last 5 minutes If one of those hosts stop sending the telemetry over the 5-minute alert window, you'll get an alert that there is no data. That's what the little checkbox means Clearly there's still historical telemetry in elasticsearch The alert me when there's no data. It's to help understand when a host container etc. Stop sending telemetry that you're trying to alert on. These are all pretty easy to test. Set up a single host... Have it sent telemetry... Set up an alert like I showed you... Stop sending telemetry and you'll get an alert Hopefully that makes sense The group by is very powerful. I would suggest looking at that But in the end if you want to set up individual alerts for every host or whatever you want to partition by, that's up to you Hopefully this makes sense I will say the method I showed you is a pretty popular way to do what we're discussing because a single alert can can be accross many host etc. If you need different threshold / condition per host then you will need to create separate alerts. ---
**Author:** ![Mike8](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike8/32/140538_2.png) [@Mike8](https://discuss.elastic.co/u/Mike8)\ **Post date:** [March 27, 2025, 4:35pm UTC](https://discuss.elastic.co/t/alert-on-no-data-per-source/376425/7 "2025-03-27T16:35:54Z")
I specifically broke the otel config on a cluster to test it: ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af3c748e52ba5700261a6c80a3a3d27528ce099e.png) This view is with query filter: `source:"cluster-a" OR source:"cluster-b"` COUNT all documents EQUATION A IS BELOW 1 FOR THE LAST 5 minutes Group alerts by: `source` `Alert me if there's no data` is checked In the dashboard you can clearly see that the 2nd cluster stopped sending data but no email was sent regarding that cluster no longer sending data. However, my related `Log Threshold` alert did trigger an email. It seems that `Alert me if there's no data` would only take effect if every single cluster stops sending data for the 5 minute period; not a single cluster/bucket. Again presumably because the UI lacks a checkbox like "consider empty buckets/doc\_count=0"; instead it just sees a general "hits = 50 so there's data" ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 27, 2025, 4:49pm UTC](https://discuss.elastic.co/t/alert-on-no-data-per-source/376425/8 "2025-03-27T16:49:31Z")
Hi @Mike8 > [@Mike8](#): > > In the dashboard you can clearly see that the 2nd cluster stopped sending data but no email was sent regarding that cluster no longer sending data. And you ran the alert .. not just tested it... > [@Mike8](#): > > `Alert me if there's no data` would only take effect if every single cluster stops sending data for the 5 minute period Hmm not my experience (nor intention, [nor what the documentation states](https://www.elastic.co/guide/en/observability/current/custom-threshold-alert.html#trigger-alert-when-no-data)) ... If I get a chance I will test again. > - **Has "group alerts by" fields** : If a previously detected group stops reporting data, a "no data" alert is triggered for the missing group.For example, consider a scenario where `host.name` is the **group alerts by** field for CPU usage above 80%. The first time the rule runs, two hosts report data: `host-1` and `host-2`. The second time the rule runs, `host-1` does not report any data, so a "no data" alert is triggered for `host-1`. When the rule runs again, if `host-1` starts reporting data again, there are a couple possible scenarios: ---
**Author:** ![Mike8](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike8/32/140538_2.png) [@Mike8](https://discuss.elastic.co/u/Mike8)\ **Post date:** [March 28, 2025, 8:10am UTC](https://discuss.elastic.co/t/alert-on-no-data-per-source/376425/9 "2025-03-28T08:10:54Z")
To clarify after looking more: The alert shows as Active in the UI but no email was sent. I assume the default setting is OK: ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/5/e56076323454dc98039a43108131b9ef940be1d1.png) Yesterday I was thinking maybe there was a strange reason of a lingering status that prevented it but it seems no. Yesterday I noticed it still showed Active because of the query previously being wrong (changing the query did not invalid the previous active alerts); although I would hope that 1 cluster being active would not block the email alert of another when using group by. I changed it to Untracked so the alert would be Recovered. But now I triggered the Alert from Recovered to Active and still no email from `Custom` (only `Log Threshold`) 😕 ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 28, 2025, 2:13pm UTC](https://discuss.elastic.co/t/alert-on-no-data-per-source/376425/10 "2025-03-28T14:13:09Z")
Aplogies having a little trouble following did you see this? If you want another email you have to add another action ![Screenshot 2025-03-28 at 7.11.57 AM](https://us1.discourse-cdn.com/elastic/original/3X/4/d/4d33231ed7eda622c7efed6279781fa7a126eec6.png) ---
**Author:** ![Mike8](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike8/32/140538_2.png) [@Mike8](https://discuss.elastic.co/u/Mike8)\ **Post date:** [March 28, 2025, 3:40pm UTC](https://discuss.elastic.co/t/alert-on-no-data-per-source/376425/11 "2025-03-28T15:40:32Z")
Ah ok thanks. It finally triggered an email when changing to "No Data". I also broke a 2nd cluster and it also triggered an email. But it's concerning that when viewing the alert details, the older broken cluster somehow had its status become "Recovered" although no logs have been sent from it for the last 2 hours. Basically can't use the Active/Recovered view as a trustworthy source ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 28, 2025, 4:10pm UTC](https://discuss.elastic.co/t/alert-on-no-data-per-source/376425/12 "2025-03-28T16:10:26Z")
> [@Mike8](#): > > But it's concerning that when viewing the alert details, the older broken cluster somehow had its status become "Recovered" although no logs have been sent from it for the last 2 hours. I had similar issues in the past where an alert as marked is recovered not because data start flowing again, but the group that it was tracking had no more data in the specific interval. Also had issues with alerts triggering too later or not triggering at all. Being honest, in my experience with Elasticsearch alerting on no data is something that is unnecessarily complicated to do. We decided to write a custom python script to trigger some ESQL queries and feedback this information into Elasticsearch to be able to create more reliable alerts. I think that I can easily replicate some issues I had to open a Github issue, but I need to find time to do that. The _false positive_ recovery would be the easiest one to replicate. ---
**Author:** ![Mike8](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike8/32/140538_2.png) [@Mike8](https://discuss.elastic.co/u/Mike8)\ **Post date:** [March 28, 2025, 4:45pm UTC](https://discuss.elastic.co/t/alert-on-no-data-per-source/376425/13 "2025-03-28T16:45:37Z")
Trying again I wasn't able to repro the status marking itself Recovered unexpectedly but the fact it happened once concerns me that I may just do 1 `Log Threshold` alert per cluster. That seems to be reliable except for my mind not wanting 6-10 identical alerts besides cluster name 😅 also it reads more logically than `Custom` threshold where the actual condition (`EQUATION BELOW 1`) does not work at all...someone in the future might see that & think the `Alert me if there's no data` is not necessary and suddenly there would be no working alerts. Whereas with `Log Threshold` the actual query is used for deciding if to trigger the alert.