What happens if primary shard fail during publication?

Elastic Stack Elasticsearch
1

As I understand ES's synchronization is kind of like 2 Phase Commit(2PC) and the problem of 2PC is when the "master" node fails during commit.

Quote from Github Reply:

An indexing request goes through the following process:

  1. Written to the translog on the primary
  2. indexed on the primary
  3. written to the translog on each replica shard
  4. indexed on each replica shard
  5. once all replicas have responded, the request returns to the client

So as long as the replicas remain alive, the change will be persisted on the replica.

So what happens if:

  1. Primary dies at phase #2 when all replicas are in old state.
  2. Primary dies at phase #4, when some replicas are in new state and some are in old state.

Thanks!

2

ES's synchronisation is not really anything like 2PC, if only because it has a single phase :slight_smile:

If the primary dies at phase 2 then a replica is promoted to primary in its place. The operation was indexed on the now-dead primary but not the replica, nor was it acked to the client.

If the primary dies at phase 4 then a replica is promoted to primary in its place. The other replicas then roll back to an earlier state and recover any missing operations from the new primary so as to be sure that they end up in the same state.

In both cases the client might see an exception, or Elasticsearch might retry the operation on the new primary and return a successful response to the client.

3

Thank you very much for clarifying the index process.

I've did some researching after post the question and learnt that ES's synchronization is a partial implementation of the PacificA algorithm. Is it correct?

Thus I'd like to confirm an additional state: Primary dies at phase #3. My assumption is:

  • A new replica is promoted as the new primary, but there is no guarantee that the request was successfully written to it.
  • Thus the client would receive exception, but the request might or might not be indexed depends on whether the translog was written to the new primary or not.

Is my understanding correct?

4

They're quite closely related.

The client might receive an exception, or Elasticsearch might retry internally on the new primary and return a successful response to the client.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.