Elasticsearch 2.3.3 : Same Query Yielding different Results with replica

Elastic Stack Elasticsearch
1

Hi,
I have seen lot of open topics in the forum which talk about the same issue but do not have a solution identified.

We have an index of about 300 million documents (500 GB disk space, 1 TB with replica) spread across 10 machines. What we observed is that a same bool query returns different results when we run the query multiple times from sense.
The top scoring document is impacted because of this inconsistency. Ex: the top score doc for a inquiry has a score of 100 and resides in shard 10. the query returns a doc with a score of 80 from shard 6 .the results completely removing shard 10 from its results (top 20) .

All the machines are identical and _cat/shards yields that all primary and replica's are in sync with exact count.
What is causing this issue ? is this a bug in 2.3.3 ?
We have tried to create a dedicated master node and inserting a sort criteria by _score in the query, but nothing solved the issue.

2

Did you try the DFS option? See https://www.elastic.co/guide/en/elasticsearch/reference/2.4/search-request-search-type.html#dfs-query-then-fetch

3

yes. my original query spans multiple indexes. the DFS query is not supported on multiple indexes, so i changed the query to point to 1 index and used DFS query, but the results are still changing.

4

May be you can use routing for your use case so you make sure that all related data are going to the same shard?

5

could you tell me little bit more about the solution ? Even if i route all my so called "related" records to a single shard, the fact that some shards are not participating in the scoring process could mean i might not see the relevant shard.
During my queries the top score varies a lot, in one of the executions i get a score of 560 for the top score but on the second run the top score comes from a different document and drops down to 480.