Running 0.12 on a 3 node cluster. Built up ~25 million documents.
Afterwards, everything was good and the match all query returned the
same number of results every time.
I did a cluster shutdown command and then started up every node. One
node got a head start, as a couple were timing out of discovery after
30 seconds. After a couple minutes I got the other two nodes started.
I am now in a green state and when i execute a : query the results
change +/- 7 results.
At no time did this cluster go split brain and the data was consistent
before cluster shutdown.
Here is my hunch:
- First node started got a head start
- Recovered a shard or shards from the gateway because it wasn't local
- Other two nodes started up and began local recovery and the local
shard must (may?) have been out of sync with the one on the gateway.
Any idea what could have happened and how to prevent this?