Different results from two clusters on different data center

Elastic Stack Elasticsearch
1

We deployed our ES cluster on two Azure data centers, and the two cluster contains exactly same index schema and documents.

The issue we found is, when querying same terms, the results from these two cluster are slightly different, event we set the primary shard to 1.

Is this expected ES behavior? What should we do to force the two cluster return exactly same results?

2

How is it different?

Any sample?

3

For a certain query, the top 2 documents from two clusters are:

    "total": 26,
    "max_score": 4.186072,
    "hits": [
        {
            "_score": 4.186072,
            "_type": "ja-jp",                
            "_source": {
                "DocumentId": "1041_1",
                "Language": "Japanese (Japan)"
            }
        },
        {
            "_score": 3.3488574,
            "_type": "ja-jp",                   
            "_source": {
                "DocumentId": "1041_2",
                "Language": "Japanese (Japan)",
            }
        }

And

    "total": 26,
    "max_score": 4.186072,
    "hits": [
        {
            "_score": 4.186072,
            "_type": "ja-jp",
            "_source": {
                "DocumentId": "1041_1",
                "Language": "Japanese (Japan)"
            }
        },
        {
            "_score": 3.3488574,
            "_type": "de-de",
            "_source": {
                "DocumentId": "1031_1",
                "Language": "German (Germany)",
            }
        }

You can see that the total document count, max score, and the score of top 2 docs are all same. But the 2nd document from two clusters are different, although the score are same.

4

Does 1031_1 also appears in the results with same score for the first cluster ?

5

Thanks reply.
Yes, you are right, the 1031_1 also appears in first cluster results (at 6th position), the score is also 3.3488574. I think that is why we got different result from two cluster.

Any solution to force two cluster always return exactly same docs with same order?

6

You can use sort to sort by _score then by another field (like a date).
If you really want to have the exact same results, you need to have exactly the same data files everywhere (also primary and replicas).

Cluster1:

Set replicas to 0
Snapshot the index
Set replicas to 1

Cluster2:

Delete old index
Restore index from the repository you created earlier
Set replicas to 1

But as soon as you will index new data, your system could diverge again.

It's not really what I'd do.

Sorting by _score only means that you don't care of anything else then the score. So even if results are different on both cluster, they are correct.

7

Sort by _score then sort by another field makes sense. Thanks David!