# The same search statement multiple times gets different total results **URL:** \ **Category:** Elasticsearch\ **Created:** [October 11, 2024, 2:06pm UTC](https://discuss.elastic.co/t/the-same-search-statement-multiple-times-gets-different-total-results/368688 "2024-10-11T14:06:27Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![deserts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deserts/32/138283_2.png) [@deserts](https://discuss.elastic.co/u/deserts)\ **Post date:** [October 11, 2024, 2:06pm UTC](https://discuss.elastic.co/t/the-same-search-statement-multiple-times-gets-different-total-results/368688/1 "2024-10-11T14:06:27Z")
I am not sure if there is a teacher who is familiar with the elasticsearch source code to help me look at this issue I raised on github. At present, I cannot get stable query results using the same conditions. The reason is that the refresh thread pool of a datanode is full. > > > \### One sentence description > > When I search using the following statement, the s…earch gets different total hits multiple times. > > \### For example: > > "hits": { > "total": { > "value": 6000000(This value is unstable and may change occasionally, but in fact the data in this index will never change from a business perspective), > "relation": "eq" > }, > ... > } > > \### Basic Information: > > Elasticsearch version: 7.17.1 > Cpu architecture: aarch64 > Jdk version: 1.8 > > Cluster overview: 9 nodes, including 6 data nodes and 3 master nodes(Server resources and jvm resources are absolutely sufficient.) > Index(my\_index) overview: 9 primary shards, 1 replica shard > > \### Search statement > > \*\*http://:9200/my\_index/\_search?request\_cache=false&explain=true&track\_total\_hits=true&q=\*\*\* > > In the above search statement, I used three key parameters, corresponding to different purposes: > track\_total\_hits=true: to get an accurate hits total value > request\_cache=false: disable query cache, eliminate the impact of cache > explain=true: explain the search, used to assist query > > In actual applications, the query has some conditions. I am using q=\* to check for unstable total doc counts. And again, the data in these problematic indexes will never change. > > In addition, when checking some basic status through /\_cluster/health and /\_cat/shards, they are all GREEN and STARTED > > It is worth mentioning that I have carefully analyzed this issue in combination with the sharing in this article: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/consistent-scoring.html#\_scores\_are\_not\_reproducible > > When I search using preference=\_shards:0,1,2,3,4,5,6,7,8, the problem still recurs randomly > > My current conclusion is that the committed status of some segments of the shards allocated on the datanode03 node of the cluster is false > > When I use preference=\_shards:0,1,2,3,4,5 to filter out the three shards 6,7,8 allocated on datanode03 (use /cat/shards/my\_index to get this information), the hits total value becomes stable > > But the most incomprehensible situation is that when observing all parameters such as os, jvm, mem, etc. through /nodes/datanode03/\_stats, no possible abnormal information is found. They all look normal. > > But the most incomprehensible situation is that I observed all the parameters such as os, jvm, mem, etc. through /nodes/datanode03/\_stats, and no possible abnormal information was found. They all looked normal. I currently only have some segments that are not committed and the shards where these segments are located are coincidentally allocated on the datanode03 node. This is a valuable piece of information. > > I want to get three help or discussion: > 1. Is /\_flush necessary? I initially judged that this action can solve this problem, but it is difficult to reproduce the phenomenon of why some segments are difficult to commit in other environments. In other words, if we don’t know why this situation occurs, we can’t avoid the problem from happening again in the future > 2. From the server level and node level, we have analyzed that the datanode03 server does not have any abnormal information, logs, etc. How can I judge whether this node really has problems? From the phenomenon of all problematic indexes, all of them are segments in the shard allocated to this node > 3. Will committed=false and searchable=false in the segment really affect the simplest query statement of \_search?request\_cache=false&explain=true&track\_total\_hits=true&q=\*? ---
**Author:** ![deserts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deserts/32/138283_2.png) [@deserts](https://discuss.elastic.co/u/deserts)\ **Post date:** [October 31, 2024, 7:44am UTC](https://discuss.elastic.co/t/the-same-search-statement-multiple-times-gets-different-total-results/368688/2 "2024-10-31T07:44:29Z")
@DavidTurner Unfortunately, this question has been raised in this forum for almost a month, and no one has paid attention to it. - ,- In addition, I don’t think that the fact that no other users have encountered this problem means that this version does not have this problem. Perhaps the ARM architecture CPU and operating system I am using are also worthy of attention? Regarding the thread stack, I used jstack to collect all the information when this problem occurred. Please give me some guidance if you have time. > [https://github.com/killersteps/jstack-dump/blob/main/jstack-2.txt](https://github.com/killersteps/jstack-dump/blob/main/jstack-2.txt) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 31, 2024, 8:04am UTC](https://discuss.elastic.co/t/the-same-search-statement-multiple-times-gets-different-total-results/368688/3 "2024-10-31T08:04:18Z")
```auto 1. Swap is not turned off on our node servers 2. We use a lot of VIRT on each node, but the memory is relatively sufficient (in other words, I don’t think that when the memory is insufficient, elasticsearch’s logic here will deadlock) ``` I saw on the GitHub issue that you have not followed [best practices and disabled swap](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html) and made sure Elasticsearch has full access to the memory of the host/VM. As you seem to have an issue with refreshing it would also help if you could provide information about the specification of the cluster, especially the type of storage used. You say that the document count does not change but the thread mentions refreshes. Are you actively updating documents in the index? Some information about the use case might help provide context. ---
**Author:** ![deserts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deserts/32/138283_2.png) [@deserts](https://discuss.elastic.co/u/deserts)\ **Post date:** [October 31, 2024, 8:32am UTC](https://discuss.elastic.co/t/the-same-search-statement-multiple-times-gets-different-total-results/368688/4 "2024-10-31T08:32:29Z")
@Christian_Dahlqvist Thank you for your reply. Indeed, not turning off swap and turning on memory locking is a deployment error. However, as we discussed before, we can suspect that this is related, but without solid evidence, we cannot conclude that this is the cause. The basic information of my environment is: Elasticsearch version: 7.17.1 Cpu architecture: aarch64 All disk types are SSD java version "1.8.0\_333" Java(TM) SE Runtime Environment (build 1.8.0\_333-b02) Java HotSpot(TM) 64-Bit Server VM (build 25.333-b02, mixed mode) Cluster overview: 9 nodes, including 6 data nodes and 3 master nodes(Server resources and jvm resources are absolutely sufficient.) Index(my\_index) overview: 9 primary shards, 1 replica shard Each index has about 10 million data items. I do not call the \_flush or \_refresh API to manually intervene in the default behavior of the cluster when writing documents. Frankly speaking, the configuration of this cluster is almost the same as the factory default configuration. To be honest, what I suspect most now is that there is an exception in the call at the operating system level, because I used strace to track the memory page fault interrupt, which seems to be related to the bytecode generated by the libjvm.so library I called. strace -o strace.txt -fp es-datanode-pid -e trace=stat,open,unlink,close ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/1/61d05e6ecd1760aabf9b8f2c8cc83c92fb80cb69.jpeg) ---
**Author:** ![deserts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deserts/32/138283_2.png) [@deserts](https://discuss.elastic.co/u/deserts)\ **Post date:** [October 31, 2024, 8:41am UTC](https://discuss.elastic.co/t/the-same-search-statement-multiple-times-gets-different-total-results/368688/6 "2024-10-31T08:41:31Z")
I don't know if there is a deadlock with the wamer thread here. The org.elasticsearch.common.cache.Cache#get(K, long, boolean) method is executed in the promote method, that is, under the else conditional branch, there is a lock in the promote method ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/b/bb2f10c542e20bd19f7e7951d23a5ddeb40da8ab.jpeg) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 31, 2024, 8:50am UTC](https://discuss.elastic.co/t/the-same-search-statement-multiple-times-gets-different-total-results/368688/7 "2024-10-31T08:50:37Z")
> [@deserts](#): > > Indeed, not turning off swap and turning on memory locking is a deployment error. However, as we discussed before, we can suspect that this is related, but without solid evidence, we cannot conclude that this is the cause. Fixing this and checking if it has any impact on the issue would allow us to confirm or rule out whether it is a contributing factor. > [@deserts](#): > > All disk types are SSD Is this local SSD or some kind of networked SSD based storage? > [@deserts](#): > > Cluster overview: 9 nodes, including 6 data nodes and 3 master nodes(Server resources and jvm resources are absolutely sufficient.) How much resources are assigned to the different node types? Is this an on-prem or cloud deployment? > [@deserts](#): > > Each index has about 10 million data items. I do not call the \_flush or \_refresh API to manually intervene in the default behavior of the cluster when writing documents. Frankly speaking, the configuration of this cluster is almost the same as the factory default configuration. Based on what you have said I assume you are updating but not adding new data to each index, is that correct? If so, how frequently are you updating? Are you using parent-child or nested documents? ---
**Author:** ![deserts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deserts/32/138283_2.png) [@deserts](https://discuss.elastic.co/u/deserts)\ **Post date:** [October 31, 2024, 9:05am UTC](https://discuss.elastic.co/t/the-same-search-statement-multiple-times-gets-different-total-results/368688/8 "2024-10-31T09:05:27Z")
@Christian_Dahlqvist I agree with your point of view. Swap and memory lock will be a key idea and direction for us to troubleshoot this problem. I'm sorry that I didn't express the information of my cluster clearly and accurately. Please allow me to explain: The disks used in our cluster are all local devices rather than network shared devices such as NAS. As mentioned before, they are all SSD disks inserted in physical servers. For the deployment architecture, we all use local physical servers for deployment, without any cloud resources (otherwise I will pull the supplier to assist us in troubleshooting from the operating system level haha) For the index, the situation is more complicated. My business creates a new index every 7 days. Generally, each new index will be written within 7 days. There are about 10 million docs. In this index in the last 7 days, I am always doing write, update, and delete operations. However, once 7 days have passed, the operation will be switched to the next index. This behavior is controlled by the client call. I have allocated 32g of heap memory to each datanode. The physical server where they are located has a 64-core cpu. There is one thing I must point out: no node in our cluster plays both masternode and datanode. However, there is a mixed deployment architecture in the cluster without problems. I am not sure if it is related to this. Finally, do you have a chat tool like telegram? I am willing to share some information with you in real time to better solve the problem (this is paid) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 31, 2024, 9:17am UTC](https://discuss.elastic.co/t/the-same-search-statement-multiple-times-gets-different-total-results/368688/9 "2024-10-31T09:17:56Z")
> [@deserts](#): > > The disks used in our cluster are all local devices rather than network shared devices such as NAS. Excellent. > [@deserts](#): > > However, once 7 days have passed, the operation will be switched to the next index. This behavior is controlled by the client call. I am assuming the inconsistency only applies to the index/indices being actively written to? Do you experience the issue if you only target indices that have refreshed and are no longer written to? > [@deserts](#): > > My business creates a new index every 7 days. Generally, each new index will be written within 7 days. There are about 10 million docs. In this index in the last 7 days, I am always doing write, update, and delete operations. It is important to note that refreshes are not synchronised across replicas, so primary and replica shards will make changes available for serach at different times. As you are adding as well as deleting you will likely see inconsistencies if some changes have refreshed on the primary but not the reoplica and vice versa. If you want to improve consustency across multiple queries I would recommend you use preference so the same set of shards are involved in the query. > [@deserts](#): > > Finally, do you have a chat tool like telegram? I am willing to share some information with you in real time to better solve the problem (this is paid) I am just a volunteer, so only log in from time to time. ---
**Author:** ![deserts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deserts/32/138283_2.png) [@deserts](https://discuss.elastic.co/u/deserts)\ **Post date:** [October 31, 2024, 9:39am UTC](https://discuss.elastic.co/t/the-same-search-statement-multiple-times-gets-different-total-results/368688/10 "2024-10-31T09:39:59Z")
@Christian_Dahlqvist We found that the abnormal situation is that when using the same query conditions for several indexes last month (from a business perspective, I can guarantee that the docs in them will not change, that is, they will not be updated, added, or deleted), different query results are obtained. After investigation, it is found that the refresh thread pool queue is full. Although I also suspected that it was related to synchronous writing and deletion, the indexes actually pointed to by these operations are the latest indexes in the past 7 days. I don’t know what impact it will have on historical indexes (I personally think it has no impact, because I saw that the searchable status of some segments of those historical indexes is false. Maybe there was a problem with the cluster at that time?) As I shared a lot of information on github, we specified preference on the client, and in the search scenario, used hash(id) to set its value, but I think this is just a means to solve the problem of stabilizing the results obtained by the same query conditions, and it is not the reason for my refresh thread pool to be full (I think there is a deadlock because the number of tasks will not decrease). In other words, even if I don’t specify preference, in an index where the data will not change again, do you think track\_total\_hits will get inconsistent totals? I think this should not be the case. ---
**Author:** ![deserts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deserts/32/138283_2.png) [@deserts](https://discuss.elastic.co/u/deserts)\ **Post date:** [October 31, 2024, 9:41am UTC](https://discuss.elastic.co/t/the-same-search-statement-multiple-times-gets-different-total-results/368688/11 "2024-10-31T09:41:46Z")
OK, I still hope to communicate with you in real time, because I am not very familiar with elasticsearch. - ，- @Christian_Dahlqvist