We are researching elasticsearch computing storage separation, and very interested in shadow replica, I would like to ask a few questions
- Under the premise of using shared storage NFS, when the primary shard loses communication with the master, after doing the primary/replica switchover, how to completely avoid the double write caused by the old primary shard continuing to write data. We understand that Lucene's own lock mechanism NativeFSLockFactory and SimpleFSLockFactory cannot be completely avoided.
- We see that the shadow replica is simply fail shadow replica to force reallocation when doing the primary/replica switch. This will cause the service to be unavailable for a short period of time. Why not consider reopening a ReadAndWriteEngine and playing back the translog?
We are also considering the implementation of the shadow replica. The replica delay based on the primary shard flush can be seen for a long time, in order to shorten the visible time of the replica. We found that Lucene's author, Michael McCandless made a simple framework for physical replication - Lucene-replicator (http://blog.mikemccandless.com/). Our main idea is that the primary shard writes data to NFS, and copies the nrt segment to the replica through the Lucene-replicator framework, and the replica is read-only.
Also, will the community accept this featrue again? We really want to contribute this featrue to the community.