# Pipeline aggregation: full histogram of an aggregation

**URL:** <https://discuss.elastic.co/t/pipeline-aggregation-full-histogram-of-an-aggregation/66167>\
**Category:** Elasticsearch\
**Created:** [November 15, 2016, 8:31pm UTC](https://discuss.elastic.co/t/pipeline-aggregation-full-histogram-of-an-aggregation/66167 "2016-11-15T20:31:21Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rodrigo\_Rezende](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rodrigo_rezende/32/4389_2.png) [@Rodrigo\_Rezende](https://discuss.elastic.co/u/Rodrigo_Rezende)\
**Post date:** [November 15, 2016, 8:31pm UTC](https://discuss.elastic.co/t/pipeline-aggregation-full-histogram-of-an-aggregation/66167/1 "2016-11-15T20:31:21Z")

</div>

Hi,

I want to estimate the distribution of an aggregation and I wonder if is there any way to perform the parent aggregation on all buckets (thing\_id) and then the percentiles over the sum so I can estimate the full histogram.

For example, I could do the following:

```
GET /data/_search
{
    "size": 0,
    "aggs" : {
        "ActionsByThing" : {
            "terms" : {
                "field" : "thing_id"
            },
            "aggs": {
                "NumberActions": {
                    "sum": {
                        "field": "nActions"
                    }
                }
            }
        },
        "PercentileOfNumberActions": {
            "percentiles_bucket": {
                "buckets_path": "ActionsByThing>NumberActions", 
                "percents": [1.0, 2.5, 5.0, 10.0, 25.0, 50.0, 75.0, 90.0, 95.0, 97.5, 99.0] 
            }
        }
    }
}

```

But that represents only the TOP K thing\_id, I'm looking for the overall distribution including the tail.

Even if I have K very large, it will be still biased.  
Even if the cardinality of thing\_id is not that large and I can afford to have K \> |thing\_id|, the aggregation response will return ActionsByThing and I really don't need it, I just care about PercentileOfNumberActions.

Is there a way to say to ES collect all ActionByThing:thing\_id just for the pipeline aggregation, but never return the ActionByThing with some hint? Maybe that would optimize things internally to consume less memory.

Any other approach?

---

<div class="post-metadata">

**Author:** ![Rodrigo\_Rezende](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rodrigo_rezende/32/4389_2.png) [@Rodrigo\_Rezende](https://discuss.elastic.co/u/Rodrigo_Rezende)\
**Post date:** [November 22, 2016, 8:02pm UTC](https://discuss.elastic.co/t/pipeline-aggregation-full-histogram-of-an-aggregation/66167/2 "2016-11-22T20:02:58Z")

</div>

is it supported?

---

<div class="post-metadata">

**Author:** ![krzysztof\_pl](https://avatars.discourse-cdn.com/v4/letter/k/eada6e/32.png) [@krzysztof\_pl](https://discuss.elastic.co/u/krzysztof_pl)\
**Post date:** [December 2, 2016, 6:24pm UTC](https://discuss.elastic.co/t/pipeline-aggregation-full-histogram-of-an-aggregation/66167/3 "2016-12-02T18:24:05Z")

</div>

hi,  
please take a look at those pages:

> **[logstash-plugins/logstash-filter-aggregate](https://github.com/logstash-plugins/logstash-filter-aggregate)**
>
> logstash-filter-aggregate - The aim of this filter is to aggregate informations available among several events (typically log lines) belonging to a same task, and finally push aggregated informatio...

  
[https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html)  
As I believe your example should be quite easy to cover by above filter-plugin.

Please note that there is a possibility to put Ruby code inside. So you are able to do a lot of things. Please remeber also that you should use only one thread (one worker). In multithreading (as you can guess) case there is a weird behaviour.

---

<div class="post-metadata">

**Author:** ![Rodrigo\_Rezende](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rodrigo_rezende/32/4389_2.png) [@Rodrigo\_Rezende](https://discuss.elastic.co/u/Rodrigo_Rezende)\
**Post date:** [December 4, 2016, 7:03pm UTC](https://discuss.elastic.co/t/pipeline-aggregation-full-histogram-of-an-aggregation/66167/4 "2016-12-04T19:03:15Z")

</div>

I appreciate your reply, but this doesn't answer my question. This has nothing to do with data ingestion / logstash. It's about [ES aggregation queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html).

---

<div class="post-metadata">

**Author:** ![Rodrigo\_Rezende](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rodrigo_rezende/32/4389_2.png) [@Rodrigo\_Rezende](https://discuss.elastic.co/u/Rodrigo_Rezende)\
**Post date:** [December 4, 2016, 7:18pm UTC](https://discuss.elastic.co/t/pipeline-aggregation-full-histogram-of-an-aggregation/66167/5 "2016-12-04T19:18:41Z")

</div>

I'm assuming now this is not supported yet and requires a new feature. Hence, opened [https://github.com/elastic/elasticsearch/issues/21962](https://github.com/elastic/elasticsearch/issues/21962)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 1, 2017, 7:19pm UTC](https://discuss.elastic.co/t/pipeline-aggregation-full-histogram-of-an-aggregation/66167/6 "2017-01-01T19:19:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.