# Pipeline aggregations on millions unique terms

**URL:** <https://discuss.elastic.co/t/pipeline-aggregations-on-millions-unique-terms/178430>\
**Category:** Elasticsearch\
**Created:** [April 25, 2019, 10:37am UTC](https://discuss.elastic.co/t/pipeline-aggregations-on-millions-unique-terms/178430 "2019-04-25T10:37:52Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yonatan\_Omer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yonatan_omer/32/42450_2.png) [@Yonatan\_Omer](https://discuss.elastic.co/u/Yonatan_Omer)\
**Post date:** [April 25, 2019, 10:37am UTC](https://discuss.elastic.co/t/pipeline-aggregations-on-millions-unique-terms/178430/1 "2019-04-25T10:37:52Z")

</div>

Our index holds log events from 100000 clients, mixed in a flat timeline, each doc looks like:  
{  
"@timestamp": "...",  
" **eventId**": "Failed to start process foo.exe",  
" **clientId**": "f68830d5-b1bf-45d6-b54b-abbf2438b709",  
}

I'm trying to write a significant terms query, which should answer:

"find unusual **eventId**'s for all **clientId**'s having a given error".

The missing part in my following query, is to convert the filter query, to the list of all **clientId** terms which contains the searched error.  
Can this be done by using pipeline aggregations, given the fact that the filter query can produce thousands of unique **clientID**'s?

Should I rather hold another index, where each doc will have all events per 'clientId' ?

```
GET /events/_search
{
  "size" : 0,
  "query" : {
    "terms" : { "eventId.keyword": ["Failed to start process dumper at"]
    }
  },
  "aggregations": {
    "top_unusual_errors": {
      "significant_terms": {
        "field": "eventId.keyword",
        "size" : 10
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [April 25, 2019, 2:29pm UTC](https://discuss.elastic.co/t/pipeline-aggregations-on-millions-unique-terms/178430/2 "2019-04-25T14:29:36Z")

</div>

> [@Yonatan\_Omer](#):
>
> Should I rather hold another index, where each doc will have all events per 'clientId' ?

Yes, that would be the more scalable approach. Example scripts and walk-through [here](https://twitter.com/elasticmark/status/1009380268409610240?s=20)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 23, 2019, 2:29pm UTC](https://discuss.elastic.co/t/pipeline-aggregations-on-millions-unique-terms/178430/3 "2019-05-23T14:29:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.