# \_source always zero in cardinality aggregation

**URL:** <https://discuss.elastic.co/t/source-always-zero-in-cardinality-aggregation/250677>\
**Category:** Elasticsearch\
**Created:** [October 1, 2020, 12:38pm UTC](https://discuss.elastic.co/t/source-always-zero-in-cardinality-aggregation/250677 "2020-10-01T12:38:01Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![autogenerated\_id](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/autogenerated_id/32/48390_2.png) [@autogenerated\_id](https://discuss.elastic.co/u/autogenerated_id)\
**Post date:** [October 1, 2020, 12:38pm UTC](https://discuss.elastic.co/t/source-always-zero-in-cardinality-aggregation/250677/1 "2020-10-01T12:38:01Z")

</div>

Hi!

I've come across something that looks like a bug. When `_score` is used in a cardinality aggregation, it always returns 0, no matter what the actual document scores are.

Here's an example that illustrates it. A `function_score` query is created and it assigns custom scores based on filters. The results are then used in two aggregations, `cardinality` and `max`.

```auto
POST score-test/_doc/1
{
  "letter": "A",
  "number": "1"
}

POST score-test/_doc/2
{
  "letter": "B",
  "number": "1"
}

POST score-test/_doc/3
{
  "letter": "A",
  "number": "2"
}

POST score-test/_doc/4
{
  "letter": "B",
  "number": "2"
}

GET score-test/_search
{
  "query": {
    "function_score": {
      "query": { "match_all": {} },
      "functions": [
        {
          "filter": { "match": { "letter": "A" } },
          "weight": 5
        },
        {
          "filter": { "match": { "number": "1" } },
          "weight": 10
        }
      ],
      "score_mode": "sum",
      "boost_mode": "replace"
    }
  },
    "aggs": {
      "test_card": {
        "cardinality": {
          "script": {
            "source": "_score"
          }
        }
      },
      "test_max": {
        "max": {
          "script": {
            "source": "_score"
        }
      }
    }
  }
}

```

Here's what it spits out when run on ES 7.4.1:

```auto
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : 15.0,
    "hits" : [
      {
        "_index" : "score-test",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 15.0,
        "_source" : {
          "letter" : "A",
          "number" : "1"
        }
      },
      {
        "_index" : "score-test",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 10.0,
        "_source" : {
          "letter" : "B",
          "number" : "1"
        }
      },
      {
        "_index" : "score-test",
        "_type" : "_doc",
        "_id" : "3",
        "_score" : 5.0,
        "_source" : {
          "letter" : "A",
          "number" : "2"
        }
      },
      {
        "_index" : "score-test",
        "_type" : "_doc",
        "_id" : "4",
        "_score" : 1.0,
        "_source" : {
          "letter" : "B",
          "number" : "2"
        }
      }
    ]
  },
  "aggregations" : {
    "test_max" : {
      "value" : 15.0
    },
    "test_card" : {
      "value" : 1
    }
  }
}

```

The document scores are fine. The `max` aggregation works as expected as well (I've tried some other metric aggs and they work, too).

However, the cardinality agg gives "1" instead of "4" (since we have 4 distinct scores). By trying more complex scripts, I've concluded that `_score` (as well as `score.doubleValue()` and the like) always return zero. This explains the "1" above (there is only one distinct value and it is 0).

Is this a bug, and if so, is there a fundamental limitation behind it?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [October 1, 2020, 1:01pm UTC](https://discuss.elastic.co/t/source-always-zero-in-cardinality-aggregation/250677/2 "2020-10-01T13:01:04Z")

</div>

Hey,

can you share the elasticsearch version you are using, as well as the **exact** query that you tried out, which includes the cardinality aggregation? I am slightly confused how you mix this with the function\_score so I would like to understand the whole thing.

Thanks!

---

<div class="post-metadata">

**Author:** ![autogenerated\_id](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/autogenerated_id/32/48390_2.png) [@autogenerated\_id](https://discuss.elastic.co/u/autogenerated_id)\
**Post date:** [October 1, 2020, 1:18pm UTC](https://discuss.elastic.co/t/source-always-zero-in-cardinality-aggregation/250677/3 "2020-10-01T13:18:28Z")

</div>

Hi,

It's ES 7.4.1, do you need more precise info?

The exact query is already in the original post (first i populate the DB with sample data, then run a search which includes the cardinality aggregation and `function_score`).

EDIT: You need to scroll down within the code block to see the query.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [October 2, 2020, 11:06am UTC](https://discuss.elastic.co/t/source-always-zero-in-cardinality-aggregation/250677/4 "2020-10-02T11:06:22Z")

</div>

Thanks, your sample now reproduces for me.

So, when running `Debug.explain(_score)` in the cardinality agg it always returns 0.0 - this is different in the min/max aggs, where the real score is used. In case, you did not know `Debug.explain()`, check [this out](https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-debugging.html)

While one could consider this a bug, I do not see a proper use-case to have the score available in the cardinality aggregation. Very curious about the use-case here. One could probably use a different kind of query and mix of aggregations to answer the same question. Some more context what you are trying to solve would be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![autogenerated\_id](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/autogenerated_id/32/48390_2.png) [@autogenerated\_id](https://discuss.elastic.co/u/autogenerated_id)\
**Post date:** [October 2, 2020, 12:22pm UTC](https://discuss.elastic.co/t/source-always-zero-in-cardinality-aggregation/250677/5 "2020-10-02T12:22:24Z")

</div>

Thanks for the help, it's much appreciated! I didn't know about `Debug.explain()`.

Let's dig into my use case step by step. Suppose an index has, among others, fields `left` and `right`. For starters, let's say we want to find out the number of distinct values in both `left` and `right` across all documents. That is, each value should be counted only once, even if it is found both in `left` and `right`. If we run two cardinality aggs and add the results together, some values may be counted twice. The only way I've found to do it (without returning all values and checking for duplicates by hand on the app side) is to run a scripted cardinality aggregation, where the script would be `[doc.left, doc.right]` (like [here](https://discuss.elastic.co/t/cardinality-aggregation-on-several-fields/34519)). This works fine.

However, the problem I'm dealing with is a bit more complex. I don't want to include all documents in the aggregation above, but `left` values based on one, and `right` values based on another criterion. So, take the index, apply `filter1` to it and pick only the `left` values of the resulting documents. Then apply `filter2` and pick only the `right` values of the documents that match. Finally, put all of these values in the same basket and find the number of distinct values.

Initially, I wanted to use named queries for this and just add an `if` to the script that would decide to return `doc.left`, `doc.right` or both, based on the matching queries. However, since this is [unsupported](https://discuss.elastic.co/t/terms-aggregation-on-named-queries/62595), I tried abusing document score to do the same thing (find out which filters matched based on the score). I failed because of the described behavior. Too bad, given that performance-wise it works fine.

I could also have two filter aggregations and compute cardinality on each of them, but then I wouldn't know how to combine the results with accounting for duplicates.

The only accurate way I can think of is returning to the application all distinct values of `left` with `filter1` applied, then doing the same for `right` and `filter2`, and finally computing the cardinality on the app side. This, however, would be too slow.

If you think there's a better way to solve this with reasonable performance, even by running multiple queries, I'd be interested to know.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [October 5, 2020, 8:33am UTC](https://discuss.elastic.co/t/source-always-zero-in-cardinality-aggregation/250677/6 "2020-10-05T08:33:26Z")

</div>

That indeed sounds tricky. I see a couple of things to take a look at, but cannot come up with the ultimate solution here.

First, have you considered adding a field `both` that contains a set of left and right at index time to not do a script on query time?

This still leaves you with the second problem of being able to filter out things. If you run the cardinality agg on `both`, even with `filter1` and `filter2` you till get wrong counts.

If every problem looks like a nail, there still is the [scripted\_metric aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/search-aggregations-metrics-scripted-metric-aggregation.html), allowing you to create your own buckets, which might help you to run a cardinality agg on - I haven't tried anything, but that may be wort a look/

---

<div class="post-metadata">

**Author:** ![autogenerated\_id](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/autogenerated_id/32/48390_2.png) [@autogenerated\_id](https://discuss.elastic.co/u/autogenerated_id)\
**Post date:** [October 5, 2020, 11:47pm UTC](https://discuss.elastic.co/t/source-always-zero-in-cardinality-aggregation/250677/7 "2020-10-05T23:47:22Z")

</div>

Thanks for the suggestions.

> First, have you considered adding a field `both` that contains a set of left and right at index time to not do a script on query time?
> 
> This still leaves you with the second problem of being able to filter out things. If you run the cardinality agg on `both` , even with `filter1` and `filter2` you till get wrong counts.

Adding `both` would be an option, but, as you're indicating, it might not bring us closer to the solution because of the second step.

> If every problem looks like a nail, there still is the [scripted\_metric aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/search-aggregations-metrics-scripted-metric-aggregation.html), allowing you to create your own buckets, which might help you to run a cardinality agg on - I haven't tried anything, but that may be wort a look/

This is interesting and might indeed work. Still, for detecing what matches `filter1` and what `filter2` in `map_script`, I guess I'd still have to use the hack with `_score` (unless you have a better idea) -- hopefully it's available in scripted metric aggregations. I can give it a shot and see what happens.

Coming back to my initial solution attempt, would it make sense to file a bug report on Github? Since `_score` is already available in similar metric aggregations, it might be an easy fix.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [October 6, 2020, 7:48am UTC](https://discuss.elastic.co/t/source-always-zero-in-cardinality-aggregation/250677/8 "2020-10-06T07:48:44Z")

</div>

As you are writing the filter yourself, you know which terms you are searching for in `filter1` or `filter2`, in the script you could just do sth like `doc['filter1'].value.contains(term)` to see if it is included in `filter1`...

And of course, please go ahead and open a github issue.

---

<div class="post-metadata">

**Author:** ![autogenerated\_id](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/autogenerated_id/32/48390_2.png) [@autogenerated\_id](https://discuss.elastic.co/u/autogenerated_id)\
**Post date:** [October 6, 2020, 11:32am UTC](https://discuss.elastic.co/t/source-always-zero-in-cardinality-aggregation/250677/9 "2020-10-06T11:32:01Z")

</div>

Unfortunately, `filter1` and `filter2` are not known ahead of time. They are generated by the app at runtime depending on the user's input, so I don't think I can easily simulate them in the script (if I understand your suggestion correctly).

I know, I know, too many constraints. 🙂 Anyway, I'll take a more detailed look at scripted aggregations.

In the meantime, I've opened an [issue](https://github.com/elastic/elasticsearch/issues/63318).

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [October 7, 2020, 6:59am UTC](https://discuss.elastic.co/t/source-always-zero-in-cardinality-aggregation/250677/10 "2020-10-07T06:59:32Z")

</div>

well, if you generate the query at runtime, you can also pass parameters to the script at runtime how your filters should look like - still worth a try IMO if you manage to find the right information before sending off the query.

---

<div class="post-metadata">

**Author:** ![autogenerated\_id](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/autogenerated_id/32/48390_2.png) [@autogenerated\_id](https://discuss.elastic.co/u/autogenerated_id)\
**Post date:** [October 7, 2020, 8:26am UTC](https://discuss.elastic.co/t/source-always-zero-in-cardinality-aggregation/250677/11 "2020-10-07T08:26:28Z")

</div>

That's correct. I'm just afraid it's impractical since the filters are not really one-liners, they get rather complex. In any case it's good to know what the options are.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [October 7, 2020, 10:59am UTC](https://discuss.elastic.co/t/source-always-zero-in-cardinality-aggregation/250677/12 "2020-10-07T10:59:02Z")

</div>

Take a look at [search templates](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/search-template.html) to help you with that and parametrize your search.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2020, 10:59am UTC](https://discuss.elastic.co/t/source-always-zero-in-cardinality-aggregation/250677/13 "2020-11-04T10:59:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.