Elasticsearch 2.3.3 : Same Query Yielding different Results with replica

I have seen lot of open topics in the forum which talk about the same issue but do not have a solution identified.

We have an index of about 300 million documents (500 GB disk space, 1 TB with replica) spread across 10 machines. What we observed is that a same bool query returns different results when we run the query multiple times from sense.
The top scoring document is impacted because of this inconsistency. Ex: the top score doc for a inquiry has a score of 100 and resides in shard 10. the query returns a doc with a score of 80 from shard 6 .the results completely removing shard 10 from its results (top 20) .

All the machines are identical and _cat/shards yields that all primary and replica's are in sync with exact count.
What is causing this issue ? is this a bug in 2.3.3 ?
We have tried to create a dedicated master node and inserting a sort criteria by _score in the query, but nothing solved the issue.

Did you try the DFS option? See https://www.elastic.co/guide/en/elasticsearch/reference/2.4/search-request-search-type.html#dfs-query-then-fetch

yes. my original query spans multiple indexes. the DFS query is not supported on multiple indexes, so i changed the query to point to 1 index and used DFS query, but the results are still changing.

May be you can use routing for your use case so you make sure that all related data are going to the same shard?

could you tell me little bit more about the solution ? Even if i route all my so called "related" records to a single shard, the fact that some shards are not participating in the scoring process could mean i might not see the relevant shard.
During my queries the top score varies a lot, in one of the executions i get a score of 560 for the top score but on the second run the top score comes from a different document and drops down to 480.