# Metric Threshold Alert reporting incorrect document count **URL:** \ **Category:** Elastic Observability\ **Created:** [November 6, 2023, 6:57pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553 "2023-11-06T18:57:43Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![vsabado](https://avatars.discourse-cdn.com/v4/letter/v/f6c823/32.png) [@vsabado](https://discuss.elastic.co/u/vsabado)\ **Post date:** [November 6, 2023, 6:57pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/1 "2023-11-06T18:57:43Z")
I have a metric threshold alert that will trigger when document count is above 30. This alert seems to trigger just fine. For the body I'm setting this: ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/2/3202eb91022f9b43e0db5b2047f079b1325a8a98.png) And this is the data that I get back when the alert fires up: {"alertId":"SOMEMADEUPID","alertName":"Aborted Alert","spaceId":"default","tags":["Dev"],"alertInstanceId":"US0418,TPASDEMO","alertActionGroup":"metrics.threshold.fired","alertActionGroupName":"Alert","context":{"group":"US0418,TPASDEMO","alertState":"ALERT","reason":"Document count is 62,474 in the last 2 hrs for US0418,TPASDEMO. Alert when \> 30.","viewInAppUrl":"SOMEURL","timestamp":"2023-11-01T19:29:19.858Z","value":{"condition0":"62,474"},"threshold":{"condition0":["30"]},"metric":{}},"date":"2023-11-01T19:29:23.552Z","state":{"start":"2023-10-17T17:54:53.840Z","duration":"1301364441000000"},"kibanaBaseUrl":"SOMEBASEURL","params":{"criteria":[{"comparator":"\>","timeSize":2,"aggType":"count","threshold":[30],"timeUnit":"h"}],"sourceId":"default","alertOnNoData":true,"alertOnGroupDisappear":true,"groupBy":["labels.storeName","labels.retailer"],"filterQueryText":"labels.http\_route: "/pos/order/{orderId}/{version}/void" and url.path : \*"},"rule":{"id":"SOMEID","name":"Aborted Alert","type":"metrics.alert.threshold","spaceId":"default","tags":["Dev"]},"alert":{"id":"US0418,TPASDEMO","actionGroup":"metrics.threshold.fired","actionGroupName":"Alert"}} The document count is an unusually large and incorrect number. Haven't quite understood why it's reporting that number and where it's coming from. Any insight would be appreciated ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 6, 2023, 7:00pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/2 "2023-11-06T19:00:46Z")
Hi @vsabado You need to share the entire Alert Configuration so we can perhaps help. Metrics create many documents ... so a document count on metrics may not be what you expect. The Way I debug these is go to discover with the same criteria and compare side by side. ---
**Author:** ![vsabado](https://avatars.discourse-cdn.com/v4/letter/v/f6c823/32.png) [@vsabado](https://discuss.elastic.co/u/vsabado)\ **Post date:** [November 6, 2023, 7:09pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/3 "2023-11-06T19:09:24Z")
Here's my complete setup for this alert: ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/c/1cc1ccfb82f71243405ce0b2fe44d96449477212.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/6/26603ce4355a36410c30f45d5fa743cc73c21c92.png) Under Observability -\> Infrastructure -\> Settings -\> Indices -\> Metric Indices, I have this value: (rum-data-view)_,traces-apm_,apm-_,logs-apm_,apm-_,metrics-apm_,apm-\* When I go plugging in that same filter into the Discover page, I get this: ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/f/5f17d6e82389261f7f3eb60d90f37c519b5d5870.png) So no explanation as to why I'm getting such a huge document count. Please let me know any further information that I can provide ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 6, 2023, 7:11pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/4 "2023-11-06T19:11:37Z")
What version of the Stack? ---
**Author:** ![vsabado](https://avatars.discourse-cdn.com/v4/letter/v/f6c823/32.png) [@vsabado](https://discuss.elastic.co/u/vsabado)\ **Post date:** [November 6, 2023, 7:12pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/5 "2023-11-06T19:12:56Z")
We are on version 8.5.2 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/0/2052f66c7a4fd23f0b4b954f04fc60878f6f4a18.png) ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 6, 2023, 7:13pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/6 "2023-11-06T19:13:05Z")
> [@vsabado](#): > > Under Observability -\> Infrastructure -\> Settings -\> Indices -\> Metric Indices, I have this value: > > (rum-data-view)_,traces-apm_,apm-_,logs-apm_,apm-_,metrics-apm_,apm-\* Show me this exactly as a screen shot... `(rum-data-view)*` that concerns me... That setting is very syntax sensitive ... Exactly which index pattern do you expect that alert data to come from. I would debug by JUST setting that as the only index in that Metrics Indices and test again... ![Screenshot 2023-11-06 at 11.15.00 AM](https://us1.discourse-cdn.com/elastic/original/3X/2/e/2e1fe0cd29f5c6ebbb1cd9942583b43a072a95da.png) ---
**Author:** ![vsabado](https://avatars.discourse-cdn.com/v4/letter/v/f6c823/32.png) [@vsabado](https://discuss.elastic.co/u/vsabado)\ **Post date:** [November 6, 2023, 7:19pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/7 "2023-11-06T19:19:01Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/2/6/2605e120e80adb889a6b8aa588d7a139624b6b3e.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/2/62d3aef132974a1ade9d6c17fc4cd1f4e82e7843.png) Not getting any data now. At least before, there was data populated in the graph ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 6, 2023, 7:23pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/8 "2023-11-06T19:23:59Z")
Where did you get that data view name... something not right (I think) Go to Discover and find out what Data Stream your documents are in... Then try that... ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 6, 2023, 7:29pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/9 "2023-11-06T19:29:14Z")
Huh I see that in a few places can you just try `traces-apm*` I am checking internally on `(rum-data-view)* ` that does not look correct, but I found it on one of my clusters too! ---
**Author:** ![vsabado](https://avatars.discourse-cdn.com/v4/letter/v/f6c823/32.png) [@vsabado](https://discuss.elastic.co/u/vsabado)\ **Post date:** [November 6, 2023, 7:37pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/10 "2023-11-06T19:37:42Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/d/c/dc59f0eee000e3d1e8f59cfac6bb5f514d7f785b.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/0/40c362972bd0ad5696cdc31cbccc18872f68ffa1.png) The graph is still showing data so that's a good sign. I'll get the alert to fire up and report back on what it says the document count is ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 6, 2023, 7:40pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/11 "2023-11-06T19:40:22Z")
Yeah \*\*\* I THINK\*\*\* that is a bug / not a valid name I think I would take that out of the Data View and that metrics indices... ---
**Author:** ![vsabado](https://avatars.discourse-cdn.com/v4/letter/v/f6c823/32.png) [@vsabado](https://discuss.elastic.co/u/vsabado)\ **Post date:** [November 6, 2023, 8:03pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/12 "2023-11-06T20:03:14Z")
Okay, I got it 4 times the last 20 mins. This rule checks every 5 mins Document count is 5,939 in the last 2 days for US0418,TPASDEMO. Alert when \> 1 Document count is 7,669 in the last 2 days for US0418,TPASDEMO. Alert when \> 1 Document count is 8,065 in the last 2 days for US0418,TPASDEMO. Alert when \> 1 Document count is 8,113 in the last 2 days for US0418,TPASDEMO. Alert when \> 1 That is really, really strange. The document hit is trending upwards, but it's changing by too much each time. This is just with traces-apm\* ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 6, 2023, 8:06pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/13 "2023-11-06T20:06:33Z")
I think your group by is not working the way you think ... it is an `OR` not an `AND` try a single group by ---
**Author:** ![vsabado](https://avatars.discourse-cdn.com/v4/letter/v/f6c823/32.png) [@vsabado](https://discuss.elastic.co/u/vsabado)\ **Post date:** [November 6, 2023, 8:14pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/14 "2023-11-06T20:14:09Z")
Hm didn't know that. How would I make it behave as AND instead of OR? That's a pivotal functionality for us for filtering out the large data that we have. So filtering by just one gives me this: Document count is 392,244 in the last 2 days for US0418. Alert when \> 1. ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/2/c24f60fbec53707550555b1796a8fddf39057fe2.png) Seems like half but that's still far too much what I really have ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 6, 2023, 8:22pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/15 "2023-11-06T20:22:15Z")
Those charts can be a bit missleading, try the actual alert... There is a fix/workaround for the "AND" if you need it .... Plus I commend you for your attempted work around with the metrics alert... soon I think there will get a "generic" alert to let you do everything you are trying but that will be in an upcoming release (you are a ways behind... 8.5.x) BTW I need to triple check the Group By... the explanation is missleading ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 6, 2023, 8:25pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/16 "2023-11-06T20:25:46Z")
BTW Did you try Logs Threshold? ![Screenshot 2023-11-06 at 12.24.51 PM](https://us1.discourse-cdn.com/elastic/original/3X/3/9/39e9cbbac52a3d6b557f812e3187eff033959ac2.png) I need to check the Group By... but won't be able to do that now > [@vsabado](#): > > ,"groupBy":["labels.storeName","labels.retailer"], @vsabado Looks like the `GROUP BY` **are ANDed** after all I am seeing Unique Combination of the 2 groups by I have done!!! So you were right! My Test Was `host.name` and `kubernetes.namespace` ```auto 4587 log entries in the last 5 mins for gke-stephen-brown-gke-dev-larger-pool-17282d5a-j2a3, recommendation. Alert when > 75. log-test is active. gke-stephen-brown-gke-dev-larger-pool-17282d5a-j2a3, recommendation - 4587 log entries have matched the following conditions: host.name does not equal vader ``` ```auto 1862 log entries in the last 5 mins for gke-stephen-brown-gke-dev-larger-pool-17282d5a-j2a3, ad. Alert when > 75. log-test is active. gke-stephen-brown-gke-dev-larger-pool-17282d5a-j2a3, ad - 1862 log entries have matched the following conditions: host.name does not equal vader ``` That is the same host with 2 different namespaces so we are good to go on that... apologies for the confusion Still not sure what is going on with yours.... got to step away. ---
**Author:** ![vsabado](https://avatars.discourse-cdn.com/v4/letter/v/f6c823/32.png) [@vsabado](https://discuss.elastic.co/u/vsabado)\ **Post date:** [November 7, 2023, 3:14pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/17 "2023-11-07T15:14:12Z")
Good morning Stepehen, Thank you so much for confirming that! Relieved to see that it's functioning as we had hoped. I just set up a log threshold alert. I'll try it now and report back ---
**Author:** ![vsabado](https://avatars.discourse-cdn.com/v4/letter/v/f6c823/32.png) [@vsabado](https://discuss.elastic.co/u/vsabado)\ **Post date:** [November 7, 2023, 3:53pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/18 "2023-11-07T15:53:49Z")
Got no alert coming from the log alert. But I'm seeing something really funky now. Alerts that really shouldn't be firing are firing. This is my settings on the discover page: ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/e/de36df2236ebea81c2981ce6d3e905c6695acacf.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/0/407b8ac7ddf4b7aaef4792f665d2db0f06e82d20.png) I have 9 hits for this the last hour, which is accurate. However metric threshold rule for this exact filter is firing off even though it's only supposed to be \> 10. My metric indices is exactly the same ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/d/5da337d18141e13be307534f708ae98101663e2d.png) I'm not sure why I'm seeing a mismatching behavior here ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 7, 2023, 10:18pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/19 "2023-11-07T22:18:12Z")
@vsabado I guess [this](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/9) got lost along the way .... I am all but positive that the `(rum-data-view)*` Is a bug and some part related to the inconsistency you are seeing. Take it out....of the data view and those settings ---
**Author:** ![vsabado](https://avatars.discourse-cdn.com/v4/letter/v/f6c823/32.png) [@vsabado](https://discuss.elastic.co/u/vsabado)\ **Post date:** [November 7, 2023, 11:13pm UTC](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553/20 "2023-11-07T23:13:46Z")
Hey @stephenb, I stripped it down to just traces-apm\* and still the same result. Works fine on my discover page though with the same data view. [Next page](https://discuss.elastic.co/t/metric-threshold-alert-reporting-incorrect-document-count/346553.md?page=2)