I am missing documents from some shard replicas, looking for advice on how to debug this further. The initial symptom is that an update on a document id failed, further investigation showed that some nodes could search for the id and some failed. For example (all identifiers changed to protect the innocent):
GET /myindex/_search?q=doc1234
returns this when the query hits the shard where the doc doesn't exist:
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "myindex",
"_type" : "mydoctype",
"_id" : "doc1234"
"_score" : 1.0,
"_source" : {
"explain" : true
}
} ]
When I add the 'explain' parameter I can figure out which shard instance it is, and turns out that the primary has 3 fewer documents than the replica.
After further investigation I see that the document count between primary and replicas are different in 469 of my 8662 shards. I suspect there's more but short of comparing doc id's between the primary/replica of each shard (not really possible given the size) that's all I can go by for now.
Running Elasticsearch v2.3.3. Some indices were brought over from v1.7 but many were created in v2.3.3 recently.
This smells similar to the discussion on this thread: How to fix primary-replica inconsistency?
Any suggestions/ideas?