ES 6.3.2 Rollup avg metric does a sum instead?

Elastic Stack Elasticsearch
1

Hi Team,

I think either there is a bug or I'm doing something silly.

This is my rollup job GET /_xpack/rollup/job/my-job/?pretty:

{
  "jobs" : [
    {
      "config" : {
        "id" : "my-job",
        "index_pattern" : "my pattern",
        "rollup_index" : "my rollup index",
        "cron" : "0 * * * * ?",
        "groups" : {
          "date_histogram" : {
            "interval" : "10m",
            "field" : "@timestamp",
            "delay" : "2d",
            "time_zone" : "UTC"
          },
          "terms" : {
            "fields" : [
              "grid_instance",
              "host",
              "host_data_center",
              "host_region"
            ]
          }
        },
        "metrics" : [
          {
            "field" : "temperature",
            "metrics" : [
              "min",
              "max",
              "avg"
            ]
          },
          {
            "field" : "memory_used_MB",
            "metrics" : [
              "min",
              "max",
              "avg"
            ]
          },
          {
            "field" : "power_draw_W",
            "metrics" : [
              "min",
              "max",
              "avg"
            ]
          },
          {
            "field" : "utilization_gpu_percent",
            "metrics" : [
              "min",
              "max",
              "avg"
            ]
          }
        ],
        "timeout" : "20s",
        "page_size" : 100000
      },

      ... ETC ...

    }
  ]
}

But I can see the aggregated value for avg looks like a sum?

image

Am I doing something wrong?

Thanks for your help,

2

Nope, you're doing everything correct! That's how Rollup represents averages internally. You'll notice there is both a value and a count for each average field. E.g.

memory_used_MB.avg._count
memory_used_MB.avg.value

At query time, we use the count and value (which is really a sum as you noticed) to rebuild the average (sum / count == average).

We do this because we want to allow averages over any interval greater-than-or-equal to the configured interval. Averaging averages together is a bad idea, so we store the sum + count instead which gives us interval freedom. We can just sum up the sums, sum up the counts, then perform the average as required.

That's part of the reason there's a separate _rollup_search endpoint, we do these sort of gymnastics behind the scenes so the user doesn't need to think about it :slight_smile:

1 Like
3

Many thanks, it makes total sense now. Great new feature.

4

Happy to help! Thanks for trying out Rollup! :slight_smile:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.