# Derivative aggregation to convert from counter to rate value

**URL:** <https://discuss.elastic.co/t/derivative-aggregation-to-convert-from-counter-to-rate-value/50965>\
**Category:** Elasticsearch\
**Created:** [May 25, 2016, 4:28pm UTC](https://discuss.elastic.co/t/derivative-aggregation-to-convert-from-counter-to-rate-value/50965 "2016-05-25T16:28:54Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bertrand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bertrand/32/45679_2.png) [@Bertrand](https://discuss.elastic.co/u/Bertrand)\
**Post date:** [May 25, 2016, 4:28pm UTC](https://discuss.elastic.co/t/derivative-aggregation-to-convert-from-counter-to-rate-value/50965/1 "2016-05-25T16:28:54Z")

</div>

Hi everyone,

We are using _collectd_ to gather NIC stats from multiple hosts and ship them to our ES cluster using _logstash_. The collected data looks as follows:

```auto
{"@timestamp":"2016-05-25T10:00:30.000+0100", "host":"host1", "collectd_type":"if_octets", "rx": 1000}
{"@timestamp":"2016-05-25T10:00:30.000+0100", "host":"host2", "collectd_type":"if_octets", "rx": 2000}
{"@timestamp":"2016-05-25T10:00:40.000+0100", "host":"host1", "collectd_type":"if_octets", "rx": 1000}

```

Data is received every 10 seconds, one record per host. The `rx` field is an ever increasing counter representing the total amount of bytes received by the NIC since the machine is alive. Nothing new here for those familiar with _collectd_...

We'd like to draw a graph representing the bandwidth (bytes per seconds) used per host over time. For this we use an _histogram_ aggregation followed by an a `derivative` to compute the amount of bytes received during the _bucket_ period. The `aggs` part of the query looks as follows (only the relevant parts are included):

```auto
"aggs": {
    "4": {
      "terms": { "field": "host" },
      "aggs": {
        "2": {
          "date_histogram": {
            "interval": "10s",
            "field": "@timestamp"
          },
          "aggs": {
            "1": { "max": { "field": "rx" } },
            "3": { "derivative": { "buckets_path": "1", "unit":"second" } }
          }
       }
    }
}

```

As I understand, here is what happens:

- the `terms` aggregation creates a bucket for each host with matching documents
- the `date_histogram` aggregation further splits every _host_ buckets into smaller buckets of _10s_ duration based on the `@timestamp` field
- for each 10s period (and for each host), the `max` function retains only the maximum `rx` value found in the time range. Since the value is an increasing counter, we could have used the `min` as well...
- finally, the `derivative` returns the difference of the max value between two consecutive buckets - which is the amount of bytes received during that period.

(stop me here if I'm wrong...)

So far so good. That query gives a breakdown of the used bandwidth **per host**. Now the question: _how should I write the query to have the **total** bandwidth of **all** hosts_ ?

Removing the `terms`aggregation on the `host` field is not correct as it will break the _max_ and _derivative_ portion responsible to convert the counter value into a rate. As far as I understand, I should be able to build something like:

- create buckets from the `date_histogram`
- split each bucket per host
- for each host, find the maximum value (or the last based on timestamp)
- compute derivative for each host, per bucket to get the amount of bytes for the time period
- then sum the total bytes of each host

Unfortunately, I have been unable to construct such query.  
Any help would be appreciated.

Thanks

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 27, 2016, 2:16am UTC](https://discuss.elastic.co/t/derivative-aggregation-to-convert-from-counter-to-rate-value/50965/2 "2016-05-27T02:16:20Z")

</div>

I think you are after this one - [https://www.elastic.co/guide/en/elasticsearch/reference/2.3/search-aggregations-pipeline-sum-bucket-aggregation.html](https://www.elastic.co/guide/en/elasticsearch/reference/2.3/search-aggregations-pipeline-sum-bucket-aggregation.html)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:48pm UTC](https://discuss.elastic.co/t/derivative-aggregation-to-convert-from-counter-to-rate-value/50965/3 "2017-07-05T22:48:40Z")

</div>