Accessing _score in the filter of aggregations

Elastic Stack Elasticsearch
1

There is an unexpected situation in how the filter aggregation works with the _score of a document. It seems that the _score is not accessible in the script inside of filter aggregation but in the same time, ES doesn't produce any errors in the response as it was in for the version 2.3 (Filter by _score in aggregation).
Below an example to reproduce this behavior:
Create index test

PUT test
{
  "settings" : {
    "number_of_shards" : 1
  },
  "mappings" : {
    "_doc" : {
      "properties" : {
        "name" : { "type" : "text" },
        "color" : { "type" : "keyword" }
      }
    }
  }
}

Create two documents, red fox and arctic fox

POST test/_doc
{
  "name": "arctic fox",
  "color": "white"
}

POST test/_doc
{
  "name": "red fox",
  "color": "red"
}

Do search for red fox and aggregate colors only of the most relevant documents

POST test/_search
{
  "query": {
    "match": {
      "name": "red fox"
    }
  },
  "aggs": {
    "mostRelevantColors": {
      "filter": {
        "script": {
          "script": {
            "source": "_score > 0.5",
            "lang": "expression"
          }
        }
      },
      "aggs": {
        "colors": {
          "terms": {
            "field": "color"
          }
        }
      }
    }
  }
}

Response

{
  "hits": {
    "hits": [
      {
        "_score": 0.87546873,
        "_type": "_doc",
        "_id": "l7petGgBlKKsTTjoJ0OO",
        "_source": {
          "color": "red",
          "name": "red fox"
        },
        "_index": "test"
      },
      {
        "_score": 0.18232156,
        "_type": "_doc",
        "_id": "f7pdtGgBlKKsTTjo-0O4",
        "_source": {
          "color": "white",
          "name": "arctic fox"
        },
        "_index": "test"
      }
    ],
    "total": 2,
    "max_score": 0.87546873
  },
  "_shards": {
    "successful": 1,
    "failed": 0,
    "skipped": 0,
    "total": 1
  },
  "took": 1,
  "aggregations": {
    "mostRelevantColors": {
      "colors": {
        "buckets": [],
        "sum_other_doc_count": 0,
        "doc_count_error_upper_bound": 0
      },
      "doc_count": 0
    }
  },
  "timed_out": false
}

I expect one color, red at the aggregations section. But there are no colors at all.

Is ES doesn't support _score in filter aggregation and silently ignore it?

UPDATE 1:
By the way, the _score is available for the stats aggregation

POST test/_search
{
  "query": {
    "match": {
      "name": "red fox"
    }
  },
  "aggs": {
      "meta": {
          "stats": {
              "script": "_score"
          }
      }
  }
}

{
  "hits": {
    "hits": [
      {
        "_score": 0.87546873,
        "_type": "_doc",
        "_id": "l7petGgBlKKsTTjoJ0OO",
        "_source": {
          "color": "red",
          "name": "red fox"
        },
        "_index": "test"
      },
      {
        "_score": 0.18232156,
        "_type": "_doc",
        "_id": "f7pdtGgBlKKsTTjo-0O4",
        "_source": {
          "color": "white",
          "name": "arctic fox"
        },
        "_index": "test"
      }
    ],
    "total": 2,
    "max_score": 0.87546873
  },
  "_shards": {
    "successful": 1,
    "failed": 0,
    "skipped": 0,
    "total": 1
  },
  "took": 8,
  "aggregations": {
    "meta": {
      "count": 2,
      "max": 0.8754687309265137,
      "sum": 1.057790294289589,
      "avg": 0.5288951471447945,
      "min": 0.18232156336307526
    }
  },
  "timed_out": false
}

We also have a doc article that said that the _score available for the aggregations.

2

After additional research (source code of ES) I've found out that the Range aggregator also has access to the _score. Below an example that solved my problem:

POST /test/_search
{
  "query": {
    "match": {
      "name": "red fox"
    }
  },
  "aggs": {
    "mostRelevantColors": {
      "range": {
          "script": {
            "source": "_score"
          },
          "ranges": [
              {
                  "from": 0.5
              }
            ]
      },
      "aggs": {
        "colors": {
          "terms": {
            "field": "color"
          }
        }
      }
    }
  }
}

Response

{
  "hits": {
    "hits": [
      {
        "_score": 0.87546873,
        "_type": "_doc",
        "_id": "l7petGgBlKKsTTjoJ0OO",
        "_source": {
          "color": "red",
          "name": "red fox"
        },
        "_index": "test"
      },
      {
        "_score": 0.18232156,
        "_type": "_doc",
        "_id": "f7pdtGgBlKKsTTjo-0O4",
        "_source": {
          "color": "white",
          "name": "arctic fox"
        },
        "_index": "test"
      }
    ],
    "total": 2,
    "max_score": 0.87546873
  },
  "_shards": {
    "successful": 1,
    "failed": 0,
    "skipped": 0,
    "total": 1
  },
  "took": 1,
  "aggregations": {
    "mostRelevantColors": {
      "buckets": [
        {
          "colors": {
            "buckets": [
              {
                "key": "red",
                "doc_count": 1
              }
            ],
            "sum_other_doc_count": 0,
            "doc_count_error_upper_bound": 0
          },
          "from": 0.5,
          "key": "0.5-*",
          "doc_count": 1
        }
      ]
    }
  },
  "timed_out": false
}
3

For this sort of “category snapping” you may be interested in looking at the significant terms and sampler aggregation: https://youtu.be/azP15yvbOBA

While your example identifies that colour:red is popular in high-scoring docs it doesn’t determine that red is significantly associated (all docs in your index might be red). Popularity is not the same as significance as the above video shows.

4

Thanks for the video! In our app we use custom scores with the ability to choose which term is more significant for the user. Despite it doesn't fit our requirements, I agree that significant terms are a good way to have a more relevance result in general.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.