Custom Aggregation [Plugin] for Elasticsearch 5.x

[varundbest (Varun Sharma)]

I'm trying to implement Weighted Avg Agg and It would be great if I could do it as a plugin, Is there any way to do so in ES 5.x? I can't seem to find any Plugins implementing Custom Aggregation logic.

[polyfractal (Zachary Tong)]

I think you could implement a weighted average using the scripted_metric aggregation: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-scripted-metric-aggregation.html

E.g. it's a metric agg that executes a series of scripts to generate the value. You could do something like this:

PUT test
{
  "mappings": {
    "doc": {
      "properties": {
        "timestamp": {
          "type": "date" 
        },
        "price": {
          "type": "integer"
        },
        "count": {
          "type": "integer"
        },
        "title": {
          "type": "keyword"
        }
      }
    }
  }
}

# Test data
POST /test/doc/_bulk
{ "index" : { "_id" : "1" } }
{"title":"bar","price":1,"count":1,"timestamp":"2017-10-01"}
{ "index" : { "_id" : "2" } }
{"title":"foo","price":2,"count":2,"timestamp":"2017-10-02"}
{ "index" : { "_id" : "3" } }
{"title":"bar","price":3,"count":3,"timestamp":"2017-10-03"}
{ "index" : { "_id" : "4" } }
{"title":"foo","price":4,"count":4,"timestamp":"2017-10-04"}
{ "index" : { "_id" : "5" } }
{"title":"bar","price":5,"count":5,"timestamp":"2017-10-05"}
{ "index" : { "_id" : "6" } }
{"title":"foo","price":6,"count":6,"timestamp":"2017-10-06"}

GET /test/_search
{
    "size": 0, 
    "query" : {
        "match_all" : {}
    },
    "aggs": {
        "weightedAvg": {
            "scripted_metric": {
                "init_script" : "params._agg.sum = 0; params._agg.count = 0;",
                "map_script" : "params._agg.sum += doc['price'].value * doc['count'].value; params._agg.count += doc['count'].value;", 
                "reduce_script" : "double finalCount = 0; double finalSum = 0; for (v in params._aggs) { finalSum += v.sum; finalCount += v.count; } return finalSum / finalCount;"
            }
        },
        "avg" : {
          "avg": {
            "field": "price"
          }
        }
    }
}

which yields:

{
  "aggregations": {
    "weightedAvg": {
      "value": 4.333333333333333
    },
    "avg": {
      "value": 3.5
    }
  }
}

The map script is executed on each document, whereas the reduce script is the final reduction of all shards together. The weighted avg itself is basically (price * count) / ∑(count)

[varundbest (Varun Sharma)]

I wish to visualize Weighted Average in Kibana Visualizations, I don't think it would be possible to do with Scripted Metric Agg.

[polyfractal (Zachary Tong)]

Yeah, Kibana wouldn't support a scripted metric agg... you'd have to write a custom visualization or something. But the same goes for a custom plugin to implement the functionality, Kibana wouldn't support that natively either.

It'd be nice to have a weighted avg function in ES natively at some point, we just have to find the time to do it