_source always zero in cardinality aggregation

Elastic Stack Elasticsearch
1

Hi!

I've come across something that looks like a bug. When _score is used in a cardinality aggregation, it always returns 0, no matter what the actual document scores are.

Here's an example that illustrates it. A function_score query is created and it assigns custom scores based on filters. The results are then used in two aggregations, cardinality and max.

POST score-test/_doc/1
{
  "letter": "A",
  "number": "1"
}

POST score-test/_doc/2
{
  "letter": "B",
  "number": "1"
}

POST score-test/_doc/3
{
  "letter": "A",
  "number": "2"
}

POST score-test/_doc/4
{
  "letter": "B",
  "number": "2"
}

GET score-test/_search
{
  "query": {
    "function_score": {
      "query": { "match_all": {} },
      "functions": [
        {
          "filter": { "match": { "letter": "A" } },
          "weight": 5
        },
        {
          "filter": { "match": { "number": "1" } },
          "weight": 10
        }
      ],
      "score_mode": "sum",
      "boost_mode": "replace"
    }
  },
    "aggs": {
      "test_card": {
        "cardinality": {
          "script": {
            "source": "_score"
          }
        }
      },
      "test_max": {
        "max": {
          "script": {
            "source": "_score"
        }
      }
    }
  }
}

Here's what it spits out when run on ES 7.4.1:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : 15.0,
    "hits" : [
      {
        "_index" : "score-test",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 15.0,
        "_source" : {
          "letter" : "A",
          "number" : "1"
        }
      },
      {
        "_index" : "score-test",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 10.0,
        "_source" : {
          "letter" : "B",
          "number" : "1"
        }
      },
      {
        "_index" : "score-test",
        "_type" : "_doc",
        "_id" : "3",
        "_score" : 5.0,
        "_source" : {
          "letter" : "A",
          "number" : "2"
        }
      },
      {
        "_index" : "score-test",
        "_type" : "_doc",
        "_id" : "4",
        "_score" : 1.0,
        "_source" : {
          "letter" : "B",
          "number" : "2"
        }
      }
    ]
  },
  "aggregations" : {
    "test_max" : {
      "value" : 15.0
    },
    "test_card" : {
      "value" : 1
    }
  }
}

The document scores are fine. The max aggregation works as expected as well (I've tried some other metric aggs and they work, too).

However, the cardinality agg gives "1" instead of "4" (since we have 4 distinct scores). By trying more complex scripts, I've concluded that _score (as well as score.doubleValue() and the like) always return zero. This explains the "1" above (there is only one distinct value and it is 0).

Is this a bug, and if so, is there a fundamental limitation behind it?

2

Hey,

can you share the elasticsearch version you are using, as well as the exact query that you tried out, which includes the cardinality aggregation? I am slightly confused how you mix this with the function_score so I would like to understand the whole thing.

Thanks!

3

Hi,

It's ES 7.4.1, do you need more precise info?

The exact query is already in the original post (first i populate the DB with sample data, then run a search which includes the cardinality aggregation and function_score).

EDIT: You need to scroll down within the code block to see the query.

4

Thanks, your sample now reproduces for me.

So, when running Debug.explain(_score) in the cardinality agg it always returns 0.0 - this is different in the min/max aggs, where the real score is used. In case, you did not know Debug.explain(), check this out

While one could consider this a bug, I do not see a proper use-case to have the score available in the cardinality aggregation. Very curious about the use-case here. One could probably use a different kind of query and mix of aggregations to answer the same question. Some more context what you are trying to solve would be greatly appreciated.

5

Thanks for the help, it's much appreciated! I didn't know about Debug.explain().

Let's dig into my use case step by step. Suppose an index has, among others, fields left and right. For starters, let's say we want to find out the number of distinct values in both left and right across all documents. That is, each value should be counted only once, even if it is found both in left and right. If we run two cardinality aggs and add the results together, some values may be counted twice. The only way I've found to do it (without returning all values and checking for duplicates by hand on the app side) is to run a scripted cardinality aggregation, where the script would be [doc.left, doc.right] (like here). This works fine.

However, the problem I'm dealing with is a bit more complex. I don't want to include all documents in the aggregation above, but left values based on one, and right values based on another criterion. So, take the index, apply filter1 to it and pick only the left values of the resulting documents. Then apply filter2 and pick only the right values of the documents that match. Finally, put all of these values in the same basket and find the number of distinct values.

Initially, I wanted to use named queries for this and just add an if to the script that would decide to return doc.left, doc.right or both, based on the matching queries. However, since this is unsupported, I tried abusing document score to do the same thing (find out which filters matched based on the score). I failed because of the described behavior. Too bad, given that performance-wise it works fine.

I could also have two filter aggregations and compute cardinality on each of them, but then I wouldn't know how to combine the results with accounting for duplicates.

The only accurate way I can think of is returning to the application all distinct values of left with filter1 applied, then doing the same for right and filter2, and finally computing the cardinality on the app side. This, however, would be too slow.

If you think there's a better way to solve this with reasonable performance, even by running multiple queries, I'd be interested to know.

6

That indeed sounds tricky. I see a couple of things to take a look at, but cannot come up with the ultimate solution here.

First, have you considered adding a field both that contains a set of left and right at index time to not do a script on query time?

This still leaves you with the second problem of being able to filter out things. If you run the cardinality agg on both, even with filter1 and filter2 you till get wrong counts.

If every problem looks like a nail, there still is the scripted_metric aggregation, allowing you to create your own buckets, which might help you to run a cardinality agg on - I haven't tried anything, but that may be wort a look/

7

Thanks for the suggestions.

First, have you considered adding a field both that contains a set of left and right at index time to not do a script on query time?

This still leaves you with the second problem of being able to filter out things. If you run the cardinality agg on both , even with filter1 and filter2 you till get wrong counts.

Adding both would be an option, but, as you're indicating, it might not bring us closer to the solution because of the second step.

If every problem looks like a nail, there still is the scripted_metric aggregation, allowing you to create your own buckets, which might help you to run a cardinality agg on - I haven't tried anything, but that may be wort a look/

This is interesting and might indeed work. Still, for detecing what matches filter1 and what filter2 in map_script, I guess I'd still have to use the hack with _score (unless you have a better idea) -- hopefully it's available in scripted metric aggregations. I can give it a shot and see what happens.

Coming back to my initial solution attempt, would it make sense to file a bug report on Github? Since _score is already available in similar metric aggregations, it might be an easy fix.

8

As you are writing the filter yourself, you know which terms you are searching for in filter1 or filter2, in the script you could just do sth like doc['filter1'].value.contains(term) to see if it is included in filter1...

And of course, please go ahead and open a github issue.

9

Unfortunately, filter1 and filter2 are not known ahead of time. They are generated by the app at runtime depending on the user's input, so I don't think I can easily simulate them in the script (if I understand your suggestion correctly).

I know, I know, too many constraints. :slight_smile: Anyway, I'll take a more detailed look at scripted aggregations.

In the meantime, I've opened an issue.

10

well, if you generate the query at runtime, you can also pass parameters to the script at runtime how your filters should look like - still worth a try IMO if you manage to find the right information before sending off the query.

11

That's correct. I'm just afraid it's impractical since the filters are not really one-liners, they get rather complex. In any case it's good to know what the options are.

12

Take a look at search templates to help you with that and parametrize your search.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.