I just performed a rolling restart of my cluster (I needed to update a plugin), following the doc. I disabled shard allocation, restarted one node, waited until it joined the cluster, reenabled allocation, waited cluster state to be green, and repeated the process on each node.
However, I was a bit surprised to see some shards being moved from one node to another after re-enabling allocation.
I thought shards were simply going to be reassigned on the node that restarted since data was already on the disk. But some of them were moved...
What is the reason for that ? Can we do something against it ?
Were you still indexing documents during the rolling restart process? If so, when you stopped a node (assuming you have replicas in addition to primary shards), the replica for some shards would have gone offline and if the node held primary shards, then the replicas for that shard in the cluster will have been promoted to primary. Now, when the node rejoins, if you were indexing, then Lucene could have merged segments, rendering your underlying segments files completely different from the shard data on the node that left the cluster to be upgraded. When that node comes back, none of the files for the shard data are the same (even though they both contain many of the same documents), so Elasticsearch does not see any reason to favor the rejoined node for allocating that shard over any other node in the cluster.
Lastly, the above problem I described will go away once sequence numbers are introduced (expected for 6.0), which will allow recovery to be based on missing index operations as opposed to file-based recovery.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.