I have seen lot of open topics in the forum which talk about the same issue but do not have a solution identified.
We have an index of about 300 million documents (500 GB disk space, 1 TB with replica) spread across 10 machines. What we observed is that a same bool query returns different results when we run the query multiple times from sense.
The top scoring document is impacted because of this inconsistency. Ex: the top score doc for a inquiry has a score of 100 and resides in shard 10. the query returns a doc with a score of 80 from shard 6 .the results completely removing shard 10 from its results (top 20) .
All the machines are identical and _cat/shards yields that all primary and replica's are in sync with exact count.
What is causing this issue ? is this a bug in 2.3.3 ?
We have tried to create a dedicated master node and inserting a sort criteria by _score in the query, but nothing solved the issue.