In-memory buffer

Elastic Stack Elasticsearch
1

Hi,
when indexing a doc, it is first indexed on primary and doc is written to in-memory buffer as well as translog buffer and then goes to replica , but then replica goes down such that quorum is not satisfied now and indexing is not successful.What happens to the doc that are written to in memory buffer and translog buffer of primary shard? are these buffers get deleted? Can someone explain it?Thanks

2

The quorum is a condition that must be satisfied before a write is attempted - see [1]

[1] https://www.elastic.co/guide/en/elasticsearch/guide/master/distrib-write.html

3

Hi Mark , Lets say quorum is satisfied before a write operation is performed on primary shard, so at this point it writes doc into in-memory buffer and translog buffer and now it hand over this process to replica phase but at this point replica goes down (such that active shards less than quorum) and writing operation is not successful. My question is what will happen to in-memory buffer of primary shard in which doc has been written previously? Whether in-memory buffer gets deleted ? or doc get deleted ? or doc remains there in buffer and when next write operation comes, new docs are appended to it?

Or is it like docs will be written to buffer only if complete indexing process(all shards of quorum) is successful? Thanks

4

As mark said the quorum check is done before we start indexing into the primary. Once the doc past the check it will not be removed. When a replica fails, as you say, the master will be notified to fail the replica and the primary will continue without it. With 2.x this means that the translog with the doc will be fsynced and we ack the write to the client. Note that the response will indicate how many successful copies were made (2.x feature). Of, course the master will also immediately start creating a new replica on another node.

5

Thanks Boaz!! can you point me to elasticsearch git code base java class where writing in buffer actually happens.

6

I'm not sure exactly what you mean. Lucene has it's own memory buffer (and is very complex). ES has a small buffer in it's BufferingTranslogWriter class on 2.x and TranslogWriter in master.

7

Yeah I was able to find it out but thanks for you reply:grinning:

8

Even better :slightly_smiling:

Sorry for the late response...