I am getting inconsistent results for some queries.
For same query , primary and replica shards are giving different number of documents.
I have 8 primary shards and 8 replica shards on 2 datanodes.
I am using routing to store and search the data.
To resolve the issue ,I deleted all the shards on one datanode. It solved the problem for some queries but some are still not resolved.
Using preference solves the consistency problem, but does not solve the root cause.
Elasticsearch version used 6.6.2.
_cat/shards output
index shard prirep state docs store node
xyz 5 r STARTED 615773 1.3gb es-data-02
xyz 5 p STARTED 615773 1.3gb es-data-01
xyz 3 r STARTED 1882378 3.9gb es-data-02
xyz 3 p STARTED 1882378 3.9gb es-data-01
xyz 7 p STARTED 20228845 38.8gb es-data-02
xyz 7 r STARTED 20228845 38.8gb es-data-01
xyz 2 p STARTED 15240307 33.8gb es-data-02
xyz 2 r STARTED 15240307 33.6gb es-data-01
xyz 4 p STARTED 8641729 13.8gb es-data-02
xyz 4 r STARTED 8641729 13.8gb es-data-01
xyz 1 r STARTED 55786281 138.1gb es-data-02
xyz 1 p STARTED 55786281 138.1gb es-data-01
xyz 6 p STARTED 3788092 6.5gb es-data-02
xyz 6 r STARTED 3788092 6.5gb es-data-01
xyz 0 r STARTED 1557571 2.6gb es-data-02
xyz 0 p STARTED 1557571 2.6gb es-data-01
This issue is happening for shard 1.
I want to know, what can be the root cause for this?
How we can avoid this behavior?