Read/write consistency

Elastic Stack Elasticsearch
1

Hello.
I guess questions about read/write consistency in Elasticsearch is a never ending story. I've been ES user for at least 7 years and still struggle with those.
My team uses Elasticsearch 6.8, we will update it eventually, but this is not an option at the moment.
Our cluster consists of 4 nodes, index has 2 shards and 2 replicas (3 total for each), all 4 nodes are master, data and ingest nodes simultaneously, not the best setup, I know.

So here's our case and additional questions I'd like to clarify.

  1. If we do update with wait_for refresh strategy what this actually means? Will response be returned once document is:
  • written and refreshed only on primary shard
  • written, refreshed on primary and written to replicas
  • written, refreshed on primary and written, refreshed on replicas
  • something else.
    Also refreshed in what way: internal or external?

  1. If we call _refresh explicitly will it refresh all shards, primary and replica across cluster before response or only shards on the node I'm making request to?

  2. We use update with wait_for on one of our services then a test polls document using GET API and receives updated state, proceeds and calls another service. That another service gets document with GET API again and receives previous version of a document and this breaks our business logic.

We tried different options for GET API request:

  • realtime=true, though it is realtime by default it uses internal refresh and as I understand may still be inconsistent on replica shard
  • refresh=true, it should have refreshed document before getting it, but still returns previous version
  • preference=_primary, we assumed that if we use wait_for refresh for update and we already got the updated document then primary shard for that document should be refreshed and consistent, but still the same problem.

We have aws ELB before all nodes to balance load and I though it might cache some responses, but my devops told me that is doesn't cache anything at all.

Thanks, all answers much appreciated.

2

  1. Written and refreshed on all in-sync shard copies (primary and in-sync replicas).

  2. All in-sync shard copies whichever node they're on.

  3. You have to wait for the update-with-wait to complete before trying to GET the doc instead of polling. Otherwise different GETs may see different versions of the doc.

3

Thank you.
I understand that we need to wait for update to complete, however we have a distributed system and update is triggered by kafka event that's why both our tests and real clients poll for result.

Is there any option we can have consistent results from polling or at least not allow dirty reads?

4

Is there any option we can have consistent results from polling or at least not allow dirty reads?

I can't think of one, sorry. You have to wait for a refresh somehow.

1 Like