# Keyword Array Field Removes Duplicates in Queries

**URL:** <https://discuss.elastic.co/t/keyword-array-field-removes-duplicates-in-queries/352582>\
**Category:** Elasticsearch\
**Tags:** painless\
**Created:** [February 6, 2024, 1:27am UTC](https://discuss.elastic.co/t/keyword-array-field-removes-duplicates-in-queries/352582 "2024-02-06T01:27:17Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![aeroplane380](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aeroplane380/32/131447_2.png) [@aeroplane380](https://discuss.elastic.co/u/aeroplane380)\
**Post date:** [February 6, 2024, 1:27am UTC](https://discuss.elastic.co/t/keyword-array-field-removes-duplicates-in-queries/352582/1 "2024-02-06T01:27:18Z")

</div>

This post relates to a problem I have encountered in my business production database, but will be described with a minimal reproducible example.

I have an index with the following mapping:

```auto
{
  "properties": {
    "test_field": {
      "type": "keyword"
    }
  }
}

```

I have two documents in the index:

```auto
{"test_field": ["good", "good", "good", "good"]}
{"test_field": ["good", "good"]}

```

I am trying to perform a search on terms in this keyword field. When I search for `good`, I want the first document to have a higher score because there are more matching terms.

As far as I understand, norms are disabled by default for keyword fields so this shouldn't affect the scoring. When I perform a `multi_match` query on this field (there are more fields in the query in my production database), both of these documents receive the same score. As the following excerpt shows, `freq` is computed as 1.

```auto
{
  "value": 0.45454544,
  "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
  "details": [{
    "value": 1,
    "description": "freq, occurrences of term within document",
    "details": []
  },
  ...

```

After investigation, it appears that the query thinks there is only one element in the field, as shown by this test query:

```auto
{
  "query": {
    "bool": {
      "must": [{
        "script_score": {
          "query": {
            "multi_match": {
              "query": "word",
              "fields": ["test_field"]
            }
          },
          "script": {
            "source": "return doc.test_field.size();"
          }
        }
      }]
    }
  }, 
  "explain": true
}

```

which returns the following

```auto
{
  "took": 39,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_shard": "[test-index][0]",
        "_node": "pJAjqOHJSP2UgyvCe_m_8g",
        "_index": "test-index",
        "_id": "EuLve40BRTvtv3Tg10Vy",
        "_score": 1.0,
        "_source": {
          "test_field": [
            "word",
            "word",
            "word",
            "word",
            "word"
          ]
        },
        "_explanation": {
          "value": 1.0,
          "description": "script score function, computed with script:\"Script{type=inline, lang='painless', idOrCode='return doc.test_field.size();', options={}, params={}}\"",
          "details": []
        }
      },
      {
        "_shard": "[test-index][0]",
        "_node": "pJAjqOHJSP2UgyvCe_m_8g",
        "_index": "test-index",
        "_id": "E-Lve40BRTvtv3Tg5kVi",
        "_score": 1.0,
        "_source": {
          "test_field": [
            "word",
            "word"
          ]
        },
        "_explanation": {
          "value": 1.0,
          "description": "script score function, computed with script:\"Script{type=inline, lang='painless', idOrCode='return doc.test_field.size();', options={}, params={}}\"",
          "details": []
        }
      }
    ]
  }
}

```

I see that the stored document still contains the duplicate strings in the field, so I am guessing that this is something that happens at index time?

Is this the correct behaviour, and if so, is there a way I can perform the query I need with the scoring I describe above?

---

<div class="post-metadata">

**Author:** ![RabBit\_BR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rabbit_br/32/82261_2.png) [@RabBit\_BR](https://discuss.elastic.co/u/RabBit_BR)\
**Post date:** [February 6, 2024, 6:46am UTC](https://discuss.elastic.co/t/keyword-array-field-removes-duplicates-in-queries/352582/2 "2024-02-06T06:46:53Z")

</div>

Hi @aeroplane380

I don't know your requirements but you could use a runtime\_mappings to retrieve the size of the array and the score function to score the docs with the highest number of elements in the array.

```auto
{
  "fields": [
    "size_array"
  ],
  "runtime_mappings": {
    "size_array": {
      "type": "long",
      "script": {
        "source": "emit(params['_source'].test_field.length)"
      }
    }
  },
  "query": {
    "function_score": {
      "query": {
        "multi_match": {
          "query": "good",
          "fields": [
            "test_field"
          ]
        }
      },
      "functions": [
        {
          "field_value_factor": {
            "field": "size_array",
            "factor": 1
          }
        }
      ]
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![aeroplane380](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aeroplane380/32/131447_2.png) [@aeroplane380](https://discuss.elastic.co/u/aeroplane380)\
**Post date:** [February 6, 2024, 9:52am UTC](https://discuss.elastic.co/t/keyword-array-field-removes-duplicates-in-queries/352582/3 "2024-02-06T09:52:24Z")

</div>

Thanks for the reply.

I actually need to score based on the number of occurrences of my search term, not on the number of elements in the array in total.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 5, 2024, 9:52am UTC](https://discuss.elastic.co/t/keyword-array-field-removes-duplicates-in-queries/352582/4 "2024-03-05T09:52:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.