# Recurring searches with the same request for dense\_vector exhibit consistency issues in the results

**URL:** <https://discuss.elastic.co/t/recurring-searches-with-the-same-request-for-dense-vector-exhibit-consistency-issues-in-the-results/372347>\
**Category:** Elasticsearch\
**Tags:** vector-search\
**Created:** [December 24, 2024, 6:22am UTC](https://discuss.elastic.co/t/recurring-searches-with-the-same-request-for-dense-vector-exhibit-consistency-issues-in-the-results/372347 "2024-12-24T06:22:29Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zona-hu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zona-hu/32/140238_2.png) [@Zona-hu](https://discuss.elastic.co/u/Zona-hu)\
**Post date:** [December 24, 2024, 6:22am UTC](https://discuss.elastic.co/t/recurring-searches-with-the-same-request-for-dense-vector-exhibit-consistency-issues-in-the-results/372347/1 "2024-12-24T06:22:30Z")

</div>

In an index without replicas, with no data being written, some vector requests, when repeated, yield inconsistent results.  
This issue is reproducible in versions 8.13.4, 8.15.1, and 8.17.0, but cannot be reproduced in version 8.7.0, indicating that there is no bug in 8.7.0.

1. create index

```auto
curl --location --request PUT 'http://elasticsearch:9200/vector_test' \
--header 'Content-Type: application/json' \
--data '{
    "mappings": {
        "dynamic": "strict",
        "properties": {
            "vector": {
                "type": "dense_vector",
                "dims": 1024,
                "index": true,
                "similarity": "cosine",
                "index_options": {
                    "type": "hnsw",
                    "m": 16,
                    "ef_construction": 100
                }
            }
        }
    },
    "settings": {
        "index": {
            "routing": {
                "allocation": {
                    "include": {
                        "_tier_preference": "data_content"
                    }
                }
            },
            "refresh_interval": "30s",
            "number_of_shards": "1",
            "number_of_replicas": "0"
        }
    }
}'

```

1. Write 10,000 random vector values and then force a \_refresh.

```auto
# -*- coding:utf-8 -*-

import json
import time

import numpy as np
import requests

REFRESH_URL = 'http://elasticsearch:9200/vector_test/_refresh'
BULK_URL = 'http://elasticsearch:9200/vector_test/_bulk'

request = requests.session()

# Generate a random vector with 1024 dimensions, where each value is a floating-point number between -1 and 1
def float32_uniform(min_value, max_value):
    random_float = np.random.uniform(min_value, max_value)
    return float(random_float)

def write():
    tmp_str = ''
    count = 0
    for id in range(10000):
        #
        vector = [float32_uniform(-1, 1) for _ in range(1024)]
        data = {'vector': vector}
        tmp_str += '{"index":{"_id":"' + str(id) + '"}}\n' + json.dumps(data) + '\n'
        if count == 1000:
            res = request.post(url=BULK_URL, headers={"Content-Type": "application/x-ndjson"}, data=tmp_str)
            print(res.text)
            tmp_str = ''
            count = 0
            time.sleep(0.2)
        count += 1
    if count != 0 and tmp_str != '':
        print(request.post(url=BULK_URL, headers={"Content-Type": "application/x-ndjson"}, data=tmp_str).json())
    request.post(REFRESH_URL)
    print("write success.")

if __name__ == ' __main__':
    write()

```

1. Begin testing to reproduce the issue. This experiment is repeated 100 times: for each iteration, a random vector is constructed and requested 100 times.

```auto
# -*- coding:utf-8 -*-

import json

import numpy as np
import requests

SEARCH_URL = 'http://elasticsearch:9200/vector_test/_search'

request = requests.session()

def float32_uniform(min_value, max_value):
    random_float = np.random.uniform(min_value, max_value)
    return float(random_float)

def request_test(loop_count, k, num_candidates):
    vector = [float32_uniform(-1, 1) for _ in range(1024)]
    body = {"from": 0, "size": 10,
            "knn": {"field": "vector", "query_vector": vector, "k": k, "num_candidates": num_candidates},
            "_source": False}
    result_dict = {}
    for i in range(loop_count):
        response = request.post(url=SEARCH_URL, json=body).json()
        hits = response['hits']['hits']
        hits_str = json.dumps(hits, ensure_ascii=False)
        if hits_str in result_dict:
            result_dict[hits_str] += 1
        else:
            result_dict[hits_str] = 1
    data_list = []
    for res, count in result_dict.items():
        data_list.append({"data": res, "count": count})

    base_count = 0
    for item in sorted(data_list, key=lambda s: s['count'], reverse=True):
        base_count = item['count']
        break
    error_count = loop_count - base_count
    print('{}/{}'.format(base_count, error_count))
    return base_count, error_count

if __name__ == ' __main__':
    success = total = 0
    for i in range(100):
        base, error_count = request_test(loop_count=100, k=10, num_candidates=20)
        success += base
        total += base + error_count
    print('{}/{}'.format(success, total))

```

Below are the test results from version 8.17.0, which show consistency issues; versions 8.13.4 and 8.15.1 also have the same problem.

 ![1](https://us1.discourse-cdn.com/elastic/original/3X/6/6/66b3b40baf0bb36cbd44781a81cc15c77a0d7cbe.png)

The following are the test results from version 8.7.0, and I have assessed the consistency to be 100%.

If the index is forcibly merged into a single segment with forcemerge, the results become stable again.

---

<div class="post-metadata">

**Author:** ![Zona-hu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zona-hu/32/140238_2.png) [@Zona-hu](https://discuss.elastic.co/u/Zona-hu)\
**Post date:** [December 24, 2024, 6:24am UTC](https://discuss.elastic.co/t/recurring-searches-with-the-same-request-for-dense-vector-exhibit-consistency-issues-in-the-results/372347/2 "2024-12-24T06:24:43Z")

</div>

> <https://github.com/elastic/elasticsearch/issues/119180>
>
> \### Elasticsearch Version
> 
> 8.17.0
> 
> \### Installed Plugins
> 
> \_No response\_
> 
> \### Jav…a Version
> 
> openjdk version "23" 2024-09-17 OpenJDK Runtime Environment (build 23+37-2369) OpenJDK 64-Bit Server VM (build 23+37-2369, mixed mode, sharing)
> 
> \### OS Version
> 
> Linux debian-002 6.1.0-18-amd64 #1 SMP PREEMPT\_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86\_64 GNU/Linux
> 
> \### Problem Description
> 
> In an index without replicas, with no data being written, some vector requests, when repeated, yield inconsistent results.
> This issue is reproducible in versions 8.13.4, 8.15.1, and 8.17.0, but cannot be reproduced in version 8.7.0, indicating that there is no bug in 8.7.0.
> 
> 
> 
> \### Steps to Reproduce
> 
> Here are the steps to reproduce the issue:
> 
> 1. Create index
> 
> \`\`\`
> curl --location --request PUT 'http://elasticsearch:9200/vector\_test' \\
> \--header 'Content-Type: application/json' \\
> \--data '{
> "mappings": {
> "dynamic": "strict",
> "properties": {
> "vector": {
> "type": "dense\_vector",
> "dims": 1024,
> "index": true,
> "similarity": "cosine",
> "index\_options": {
> "type": "hnsw",
> "m": 16,
> "ef\_construction": 100
> }
> }
> }
> },
> "settings": {
> "index": {
> "routing": {
> "allocation": {
> "include": {
> "\_tier\_preference": "data\_content"
> }
> }
> },
> "refresh\_interval": "30s",
> "number\_of\_shards": "1",
> "number\_of\_replicas": "0"
> }
> }
> }'
> \`\`\`
> 
> 2. Write 10,000 random vector values and then force a \_refresh. 
> \`\`\`
> \# -\*- coding:utf-8 -\*-
> 
> import json
> import time
> 
> import numpy as np
> import requests
> 
> REFRESH\_URL = 'http://elasticsearch:9200/vector\_test/\_refresh'
> BULK\_URL = 'http://elasticsearch:9200/vector\_test/\_bulk'
> 
> request = requests.session()
> 
> \# Generate a random vector with 1024 dimensions, where each value is a floating-point number between -1 and 1
> def float32\_uniform(min\_value, max\_value):
> random\_float = np.random.uniform(min\_value, max\_value)
> return float(random\_float)
> 
> 
> def write():
> tmp\_str = ''
> count = 0
> for id in range(10000):
> #
> vector = \[float32\_uniform(-1, 1) for \_ in range(1024)\]
> data = {'vector': vector}
> tmp\_str += '{"index":{"\_id":"' + str(id) + '"}}\\n' + json.dumps(data) + '\\n'
> if count == 1000:
> res = request.post(url=BULK\_URL, headers={"Content-Type": "application/x-ndjson"}, data=tmp\_str)
> print(res.text)
> tmp\_str = ''
> count = 0
> time.sleep(0.2)
> count += 1
> if count != 0 and tmp\_str != '':
> print(request.post(url=BULK\_URL, headers={"Content-Type": "application/x-ndjson"}, data=tmp\_str).json())
> request.post(REFRESH\_URL)
> print("write success.")
> 
> 
> if \_\_name\_\_ == '\_\_main\_\_':
> write()
> 
> \`\`\`
> 3. Begin testing to reproduce the issue. This experiment is repeated 100 times: for each iteration, a random vector is constructed and requested 100 times.
> 
> \`\`\`
> \# -\*- coding:utf-8 -\*-
> 
> import json
> 
> import numpy as np
> import requests
> 
> SEARCH\_URL = 'http://elasticsearch:9200/vector\_test/\_search'
> 
> request = requests.session()
> 
> 
> def float32\_uniform(min\_value, max\_value):
> random\_float = np.random.uniform(min\_value, max\_value)
> return float(random\_float)
> 
> 
> def request\_test(loop\_count, k, num\_candidates):
> vector = \[float32\_uniform(-1, 1) for \_ in range(1024)\]
> body = {"from": 0, "size": 10,
> "knn": {"field": "vector", "query\_vector": vector, "k": k, "num\_candidates": num\_candidates},
> "\_source": False}
> result\_dict = {}
> for i in range(loop\_count):
> response = request.post(url=SEARCH\_URL, json=body).json()
> hits = response\['hits'\]\['hits'\]
> hits\_str = json.dumps(hits, ensure\_ascii=False)
> if hits\_str in result\_dict:
> result\_dict\[hits\_str\] += 1
> else:
> result\_dict\[hits\_str\] = 1
> data\_list = \[\]
> for res, count in result\_dict.items():
> data\_list.append({"data": res, "count": count})
> 
> base\_count = 0
> for item in sorted(data\_list, key=lambda s: s\['count'\], reverse=True):
> base\_count = item\['count'\]
> break
> error\_count = loop\_count - base\_count
> print('{}/{}'.format(base\_count, error\_count))
> return base\_count, error\_count
> 
> 
> if \_\_name\_\_ == '\_\_main\_\_':
> success = total = 0
> for i in range(100):
> base, error\_count = request\_test(loop\_count=100, k=10, num\_candidates=20)
> success += base
> total += base + error\_count
> print('{}/{}'.format(success, total))
> 
> \`\`\`
> 
> Below are the test results from version 8.17.0, which show consistency issues; versions 8.13.4 and 8.15.1 also have the same problem.
> !\[Image\](https://github.com/user-attachments/assets/edc7788f-8e7d-4d20-a7b9-7d4bf2d85bcf)
> 
> 
> The following are the test results from version 8.7.0, and I have assessed the consistency to be 100%.
> !\[Image\](https://github.com/user-attachments/assets/1700e5cd-d778-4c81-a401-9e5282aa35ed)
> 
> 
> 
> \### Logs (if relevant)
> 
> \_No response\_

I've submitted an issue on GitHub, but no one has responded. Could someone please take the time to test whether my conclusion is correct?

---

<div class="post-metadata">

**Author:** ![S-Dragon0302](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s-dragon0302/32/100509_2.png) [@S-Dragon0302](https://discuss.elastic.co/u/S-Dragon0302)\
**Post date:** [April 2, 2025, 10:35am UTC](https://discuss.elastic.co/t/recurring-searches-with-the-same-request-for-dense-vector-exhibit-consistency-issues-in-the-results/372347/3 "2025-04-02T10:35:55Z")

</div>

I also encountered this problem, is there any solution?