# Suggestion on summarizing running total value **URL:** \ **Category:** Elasticsearch\ **Created:** [July 15, 2020, 5:20pm UTC](https://discuss.elastic.co/t/suggestion-on-summarizing-running-total-value/241339 "2020-07-15T17:20:14Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Jehutywong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jehutywong/32/50527_2.png) [@Jehutywong](https://discuss.elastic.co/u/Jehutywong)\ **Post date:** [July 15, 2020, 5:20pm UTC](https://discuss.elastic.co/t/suggestion-on-summarizing-running-total-value/241339/1 "2020-07-15T17:20:14Z")
ES version 7.6 I am finding difficult to do summarize or transforms on running total values like network traffic metrics. For example i want to get the max bucket on inbound bytes/s. But prior to that, looks like i have to calculate rate (out of the running total) first. A simple derivative aggregation may easily hit the too many bucket error. Especially the interval setting in date\_histogram is short (e.g 1 minute), or there is additional terms aggregation While considering transforms, it complains "Unsupported aggregation type [date\_histogram]" Please any suggestion? Thanks in advance. ---
**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\ **Post date:** [July 15, 2020, 6:48pm UTC](https://discuss.elastic.co/t/suggestion-on-summarizing-running-total-value/241339/2 "2020-07-15T18:48:13Z")
Can you provide some example of what you try to do? You can also post the aggregations and/or transform you tried. As for transform: `date_histogram` is supported as part of `group_by`, I wonder why you try to use it in the aggregation part. ---
**Author:** ![Jehutywong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jehutywong/32/50527_2.png) [@Jehutywong](https://discuss.elastic.co/u/Jehutywong)\ **Post date:** [July 16, 2020, 3:22am UTC](https://discuss.elastic.co/t/suggestion-on-summarizing-running-total-value/241339/3 "2020-07-16T03:22:56Z")
The reason i put date\_histogram in the aggs part was that i got following error, if i move the date\_histogram to the group\_by part. "reason" : "derivative aggregation [diff] must have a histogram, date\_histogram or auto\_date\_histogram as parent" **here is my pivot directive:** "pivot": { "group\_by": { "host.keyword": { "terms": { "field": "host.keyword" } }, "@timestamp": { "date\_histogram": { "field": "@timestamp", "fixed\_interval": "5m" } } }, "aggregations": { "diff": { "derivative": { "buckets\_path": "read" } }, "read": { "max": { "field": "read" } } }, "max\_page\_search\_size": 2000 }, **Here's my derivative aggs (i also tried to use partition in terms aggs. Even if i cut it into small pieces, the number of bucket will exceed limit in a couple of months time range):** "aggs": { "host": { "terms": { "field": "host.keyword", "size": 9999 }, "aggs": { "date": { "date\_histogram": { "field": "@timestamp", "fixed\_interval": "5m" }, "aggs": { "inbound": { "max": { "field": "rx" } }, "rate": { "derivative": { "buckets\_path": "inbound" } } } } } } } ---
**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\ **Post date:** [July 16, 2020, 5:15am UTC](https://discuss.elastic.co/t/suggestion-on-summarizing-running-total-value/241339/4 "2020-07-16T05:15:30Z")
I see it now. The problem with the transform: "derivative" isn't supported, too. This is a bigger technical limitation and can't be easily fixed. The other problem I see with your usecase is the amount of data you have. With aggregations you will always run into size problems for usecases like this. You need some sort of chunking/paging which either transform or composite aggs can provide (transform uses composite aggs, but again composite aggs don't support pipeline aggregations). The only solution I see at the moment is using a transform with everything you have but the derivative aggregation. For adding derivative I suggest to write some custom code that runs queries on the transform destination index, injects the derivative and writes the result back. ---
**Author:** ![Jehutywong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jehutywong/32/50527_2.png) [@Jehutywong](https://discuss.elastic.co/u/Jehutywong)\ **Post date:** [July 16, 2020, 6:27am UTC](https://discuss.elastic.co/t/suggestion-on-summarizing-running-total-value/241339/5 "2020-07-16T06:27:44Z")
So I'd have to complete the derivative out site ES. Not that decent, but a confirmation of i haven't miss any cool feature in ES is good enough. Really appreciate. @Hendrik_Muhs ---
**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\ **Post date:** [July 16, 2020, 6:47am UTC](https://discuss.elastic.co/t/suggestion-on-summarizing-running-total-value/241339/6 "2020-07-16T06:47:51Z")
Feel free to open an enhancement request, it sounds like an interesting use case. It would make sense to add something to transform to enable such things, similar to what pipeline aggregations offer. Good feedback! ---
**Author:** ![Jehutywong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jehutywong/32/50527_2.png) [@Jehutywong](https://discuss.elastic.co/u/Jehutywong)\ **Post date:** [July 16, 2020, 7:54am UTC](https://discuss.elastic.co/t/suggestion-on-summarizing-running-total-value/241339/7 "2020-07-16T07:54:54Z")
#59684 filed ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 13, 2020, 7:55am UTC](https://discuss.elastic.co/t/suggestion-on-summarizing-running-total-value/241339/8 "2020-08-13T07:55:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.