# How do I aggregate across subsearches? **URL:** \ **Category:** Kibana\ **Created:** [May 7, 2018, 11:28am UTC](https://discuss.elastic.co/t/how-do-i-aggregate-across-subsearches/130825 "2018-05-07T11:28:47Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Joshua\_Fox](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@Joshua\_Fox](https://discuss.elastic.co/u/Joshua_Fox)\ **Post date:** [May 7, 2018, 11:28am UTC](https://discuss.elastic.co/t/how-do-i-aggregate-across-subsearches/130825/1 "2018-05-07T11:28:48Z")
A certain periodic task is supposed to happen once a minute. I send an alert when this is hanging using the following condition in a Threshold Alert. ``` WHEN count() GROUPED OVER top 1 'myPeriodicTaskLog' IS BELOW 1 FOR THE LAST 2 minutes ``` My complication: This task is occurring separately in multiple Docker Instances, and I want to check that _none of them_ is blocked. I want to say "The key `myPeriodicTaskLog` must occur each minute in each instance. Otherwise send an alert." I have the field `instance_name`. Each instance's name is assigned pseudorandomly on each deployment (i.e., something like `"a58hgh12g2"`). So, I cannot code the condition to include these names as literals but can use these values to aggregate. How do I group in this way? ---
**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\ **Post date:** [May 8, 2018, 1:13pm UTC](https://discuss.elastic.co/t/how-do-i-aggregate-across-subsearches/130825/2 "2018-05-08T13:13:03Z")
Hi @Joshua_Fox, Do you happen to know the exact number of instances available? Or is that something that can change over time? ---
**Author:** ![Joshua\_Fox](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@Joshua\_Fox](https://discuss.elastic.co/u/Joshua_Fox)\ **Post date:** [May 11, 2018, 8:23am UTC](https://discuss.elastic.co/t/how-do-i-aggregate-across-subsearches/130825/3 "2018-05-11T08:23:12Z")
It is usually two but we might scale up to three. if a solution depends on having a constant number, that could be good enough. ---
**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\ **Post date:** [May 14, 2018, 3:38pm UTC](https://discuss.elastic.co/t/how-do-i-aggregate-across-subsearches/130825/4 "2018-05-14T15:38:22Z")
Check out this watch. It will ensure none of the nodes in your cluster are down (but it assumes you know the node count). This might work for you: ```auto { "trigger": { "schedule": { "interval": "1m" } }, "input": { "search": { "request": { "search_type": "query_then_fetch", "indices": [ ".monitoring-es-6*" ], "types": [], "body": { "size": 1, "sort": [ { "timestamp": { "order": "desc" } } ], "_source": "cluster_stats.nodes.count.total", "query": { "bool": { "filter": { "term": { "type": "cluster_stats" } } } } } } } }, "condition": { "compare": { "ctx.payload.hits.hits.0._source.cluster_stats.nodes.count.total": { "lt": 3 } } }, "actions": { "my-logging-action": { "logging": { "level": "error", "text": "Node is down! We only detect {{ctx.payload.hits.hits.0._source.cluster_stats.nodes.count.total}} nodes." } }, "send_email": { "email": { "profile": "standard", "to": [ "steve@steve.com" ], "subject": "Steve! A node in your cluster is down!", "body": { "text": "We only detect {{ctx.payload.hits.hits.0._source.cluster_stats.nodes.count.total}} nodes." } } }, } } ``` ---
**Author:** ![Joshua\_Fox](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@Joshua\_Fox](https://discuss.elastic.co/u/Joshua_Fox)\ **Post date:** [May 16, 2018, 1:22pm UTC](https://discuss.elastic.co/t/how-do-i-aggregate-across-subsearches/130825/6 "2018-05-16T13:22:02Z")
Thank you. That query does not mention `myPeriodicTaskLog` . So it seems that it is tracking node liveness, but not liveness of that thread on each node. Is that right? I'd like to do 1. node\_list = query for a list of nodes that have been live at all in the last minute, based on `instance_name` field which occurs in each log line 2. query for the presence of `myPeriodicTaskLog` in the last minute, _grouped by node_ for each node in node\_list. If that query does not return at least 1 value for each node, send an alert. is this feasible? ---
**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\ **Post date:** [May 16, 2018, 3:27pm UTC](https://discuss.elastic.co/t/how-do-i-aggregate-across-subsearches/130825/7 "2018-05-16T15:27:40Z")
It should be. The following watch will look at packetbeat data by grouping all data by ip address, then using that grouping, determine if any of the documents in each ip bucket are missing a response code. If so, it will fire an alert. This feels similar to what you're doing so hopefully this will help. One thing I recommend is writing an ES query that will actually detect the data you are hoping to use in the condition. If you can do that, you can definitely create a watch for it. ```auto { "trigger": { "schedule": { "interval": "10s" } }, "input": { "search": { "request": { "search_type": "query_then_fetch", "indices": [ "packetbeat-*" ], "types": [], "body": { "size": 0, "query": { "match_all": {} }, "aggs": { "unique_beat_names": { "terms": { "field": "ip", "size": 5 }, "aggs": { "response_code": { "filter": { "exists": { "field": "dns.response_code" } } } } } } } } } }, "condition": { "script": { "source": "ctx.vars.missing=false;for (def beat_name : ctx.payload.aggregations.unique_beat_names.buckets){if(beat_name.doc_count == 0 || beat_name.response_code.doc_count == 0){ctx.vars.missing=true;}}return ctx.vars.missing;", "lang": "painless" } }, "actions": { "my-logging-action": { "logging": { "level": "info", "text": "Oh yea" } } } } ``` ---
**Author:** ![Joshua\_Fox](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@Joshua\_Fox](https://discuss.elastic.co/u/Joshua_Fox)\ **Post date:** [May 17, 2018, 9:59am UTC](https://discuss.elastic.co/t/how-do-i-aggregate-across-subsearches/130825/8 "2018-05-17T09:59:46Z")
Thank you. It looks like `ctx.payload.aggregations.unique_beat_names.buckets` gives the list of unique IPs (in our case, that will be Nodes/instances by `instance_name` rather than IP.) Then , this script gives the boolean for an alert . Importantly. it looks like this is in the Painless scripting language which is rich enough to encode any needed logic ``` ctx.vars.missing = false; for (def beat_name : ctx.payload.aggregations.unique_beat_names.buckets) { if (beat_name.doc_count == 0 || beat_name.response_code.doc_count == 0) { ctx.vars.missing=true; } } return ctx.vars.missing;` ``` ---
**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\ **Post date:** [May 17, 2018, 3:16pm UTC](https://discuss.elastic.co/t/how-do-i-aggregate-across-subsearches/130825/9 "2018-05-17T15:16:15Z")
Yup! Let me know if that helps! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 14, 2018, 3:16pm UTC](https://discuss.elastic.co/t/how-do-i-aggregate-across-subsearches/130825/10 "2018-06-14T15:16:19Z")
Closed on June 14, 2018, 3:16pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.