# ES filter composite aggregation result by doc\_count

**URL:** <https://discuss.elastic.co/t/es-filter-composite-aggregation-result-by-doc-count/311727>\
**Category:** Kibana\
**Created:** [August 9, 2022, 12:40pm UTC](https://discuss.elastic.co/t/es-filter-composite-aggregation-result-by-doc-count/311727 "2022-08-09T12:40:43Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mirokrastev](https://avatars.discourse-cdn.com/v4/letter/m/f14d63/32.png) [@mirokrastev](https://discuss.elastic.co/u/mirokrastev)\
**Post date:** [August 9, 2022, 12:40pm UTC](https://discuss.elastic.co/t/es-filter-composite-aggregation-result-by-doc-count/311727/1 "2022-08-09T12:40:43Z")

</div>

I have the following ES query:

```auto
GET my-index/_search
{
  "size": 0,
  "aggs": {
    "my_bucket": {
      "composite": {
        "size": 10000,
        "sources": [
          {
            "stk1": {
              "terms": {
                "field": "calling_number.keyword"
              }
            }
          },
          {
            "stk2": {
              "terms": {
                "field": "called_number.keyword"
              }
            }
          }
        ]
      }
    }
  }
}

```

Here is the output:

```auto
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : []
  },
  "aggregations" : {
    "my_bucket" : {
      "after_key" : {
        "stk1" : "9",
        "stk2" : "99"
      },
      "buckets" : [
        {
          "key" : {
            "stk1" : "8",
            "stk2" : "99"
          },
          "doc_count" : 1
        },
        {
          "key" : {
            "stk1" : "9",
            "stk2" : "98"
          },
          "doc_count" : 1
        },
        {
          "key" : {
            "stk1" : "9",
            "stk2" : "99"
          },
          "doc_count" : 2
        }
      ]
    }
  }
}

```

I want to filter the buckets by `doc_count >= 3`, basically to exclude everything that is not \>= 3. The result from this query will be used in Watcher to monitor activities which exceedes some threshold, in my case `3`

---

<div class="post-metadata">

**Author:** ![Jonathan\_Buttner](https://avatars.discourse-cdn.com/v4/letter/j/9d8465/32.png) [@Jonathan\_Buttner](https://discuss.elastic.co/u/Jonathan_Buttner)\
**Post date:** [August 9, 2022, 12:49pm UTC](https://discuss.elastic.co/t/es-filter-composite-aggregation-result-by-doc-count/311727/2 "2022-08-09T12:49:27Z")

</div>

Hi @mirokrastev ,

Thanks for your question. I don't believe there is a way to filter composite aggregations by the `doc_count`. Here's a reference issue discussing this:

> <https://github.com/elastic/elasticsearch/issues/32452>
>
> \*\*Elasticsearch version\*\* : docker.elastic.co/elasticsearch/elasticsearch:6.3.1
> …
> 
> \*\*Plugins installed\*\*: \[\]
> 
> \*\*JVM version\*\* the one from the docker-image (docker.elastic.co/elasticsearch/elasticsearch:6.3.1):
> 
> \*\*OS version\*\* like above (docker.elastic.co/elasticsearch/elasticsearch:6.3.1):
> 
> \*\*Description of the problem including expected versus actual behavior\*\*:
> 
> I am experiencing the error "unknown field \[min\_doc\_count\]" on a terms-aggregation. I placed said terms-aggregation as the single source in a composite-aggregation like so:
> 
> 
> \`\`\`
> {
> "aggs" : {
> "my\_buckets": {
> "composite" : {
> "sources" : \[
> { "product": { "terms" : { "field": "product", "min\_doc\_count": 2 } } }
> \]
> }
> }
> }
> 
> \`\`\`
> 
> 
> (I did that because I wanted to use the pagination-feature of composite-aggregations)
> 
> The aggregation works fine on it's own. Only when I put it in a composite-aggregation it fails.
> 
> I have asked an according question on SO:
> 
> \[https://stackoverflow.com/questions/51557626/elasticsearch-unknown-field-min-doc-count\](https://stackoverflow.com/questions/51557626/elasticsearch-unknown-field-min-doc-count)
> 
>  
> 
> \*\*Steps to reproduce\*\*:
> (see query above)
> 
> 
> I hope this is just a bug, although I fear that maybe I just did not understand composite-queries...
> 
> Hope anybody can help?
> 
> Thanks!

Terms aggregations does support this using the `min_doc_count`: [Terms aggregation | Elasticsearch Guide [8.3] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_minimum_document_count_4)

Would a terms aggregation work for your use case?

---

<div class="post-metadata">

**Author:** ![mirokrastev](https://avatars.discourse-cdn.com/v4/letter/m/f14d63/32.png) [@mirokrastev](https://discuss.elastic.co/u/mirokrastev)\
**Post date:** [August 9, 2022, 1:13pm UTC](https://discuss.elastic.co/t/es-filter-composite-aggregation-result-by-doc-count/311727/3 "2022-08-09T13:13:48Z")

</div>

Hello @Jonathan_Buttner,

Thanks for the fast answer. Here is the idea that I want to accomplish:

I want to get the `calling_numbers` which calls another `called_number` more than X times (in my case 3 times) in the span of Y hours (in my case 1 hour).  
Terms aggregation will not work, at least not with the approach that I have taken, because one `calling_number`, for example 99, could have called `called_number` "98" 2 times, and `called_number` "100" 2 times, and it will not exceed the threshold of 3 to the same `called_number`.

Do you have any idea how I may accomplish this task? The result will be used inside a watcher.

---

<div class="post-metadata">

**Author:** ![Jonathan\_Buttner](https://avatars.discourse-cdn.com/v4/letter/j/9d8465/32.png) [@Jonathan\_Buttner](https://discuss.elastic.co/u/Jonathan_Buttner)\
**Post date:** [August 9, 2022, 7:29pm UTC](https://discuss.elastic.co/t/es-filter-composite-aggregation-result-by-doc-count/311727/4 "2022-08-09T19:29:28Z")

</div>

I'm not super familiar with watcher and sorry if I'm misunderstanding how the data is formatted but here's an example of how I think a nested terms aggregation could accomplish this:

> **Example**
>
> ```auto
> POST my-index-000001/_doc/
> {
> "calling_number": "1",
> "called_number": "50"
> }
> 
> POST my-index-000001/_doc/
> {
> "calling_number": "2",
> "called_number": "60"
> }
> 
> POST my-index-000001/_doc/
> {
> "calling_number": "1",
> "called_number": "51"
> }
> 
> POST my-index-000001/_doc/
> {
> "calling_number": "1",
> "called_number": "50"
> }
> 
> POST _search
> {
> "size": 0, 
> "aggs": {
> "calling": {
> "terms": {
> "field": "calling_number.keyword"
> },
> "aggs": {
> "called": {
> "terms": {
> "field": "called_number.keyword",
> "min_doc_count": 2
> }
> }
> }
> }
> }
> }
> 
> ```

> **Results**
>
> ```auto
> {
> "took": 1,
> "timed_out": false,
> "_shards": {
> "total": 1,
> "successful": 1,
> "skipped": 0,
> "failed": 0
> },
> "hits": {
> "total": {
> "value": 4,
> "relation": "eq"
> },
> "max_score": null,
> "hits": []
> },
> "aggregations": {
> "calling": {
> "doc_count_error_upper_bound": 0,
> "sum_other_doc_count": 0,
> "buckets": [
> {
> "key": "1",
> "doc_count": 3,
> "called": {
> "doc_count_error_upper_bound": 0,
> "sum_other_doc_count": 0,
> "buckets": [
> {
> "key": "50",
> "doc_count": 2
> }
> ]
> }
> },
> {
> "key": "2",
> "doc_count": 1,
> "called": {
> "doc_count_error_upper_bound": 0,
> "sum_other_doc_count": 0,
> "buckets": []
> }
> }
> ]
> }
> }
> }
> 
> ```

If you remove the `"min_doc_count": 2` the results would look like:

> **No minimum**
>
> ```auto
> {
> "took": 1,
> "timed_out": false,
> "_shards": {
> "total": 1,
> "successful": 1,
> "skipped": 0,
> "failed": 0
> },
> "hits": {
> "total": {
> "value": 4,
> "relation": "eq"
> },
> "max_score": null,
> "hits": []
> },
> "aggregations": {
> "calling": {
> "doc_count_error_upper_bound": 0,
> "sum_other_doc_count": 0,
> "buckets": [
> {
> "key": "1",
> "doc_count": 3,
> "called": {
> "doc_count_error_upper_bound": 0,
> "sum_other_doc_count": 0,
> "buckets": [
> {
> "key": "50",
> "doc_count": 2
> },
> {
> "key": "51",
> "doc_count": 1
> }
> ]
> }
> },
> {
> "key": "2",
> "doc_count": 1,
> "called": {
> "doc_count_error_upper_bound": 0,
> "sum_other_doc_count": 0,
> "buckets": [
> {
> "key": "60",
> "doc_count": 1
> }
> ]
> }
> }
> ]
> }
> }
> }
> 
> ```

So it'll only return the `called` buckets if they meet the minimum required count.

The `in the span of Y hours` will be more difficult to achieve, I would look into the pipeline aggregations, maybe a sliding window would work. If not, it'd probably need to be done by the server that is doing the query for the data.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 6, 2022, 7:30pm UTC](https://discuss.elastic.co/t/es-filter-composite-aggregation-result-by-doc-count/311727/5 "2022-09-06T19:30:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.