When data can be lost?

[AdamLepkowski (Adam Łepkowski)]

Hi, I’m indexing my data to the Elasticsearch using JDBC importer tool. I know that data is not immediately saved on a hard drive but they are stored in a memory buffer and there is possibility data can be lost? How many possibilities exist that data can be lost? How do you know that you lost your data? Is there any way to secure it? How do you supplement it again – night load?

We ca assume that I have 3 nodes with the elasticsearch 5.x

[jasontedor]

@AdamLepkowski

I know that data is not immediately saved on a hard drive but they are stored in a memory buffer and there is possibility data can be lost?

This is not quite correct. If the client receives a successful acknowledgement from Elasticsearch and you have not changed the default translog durability settings, then before the returning the successful acknowledgement all shards will have written the request into the translog and fsynced the translog to disk. In the case of a crash, the request will be replayed from the translog.

In short, you can assume that acknowledged operations are safely on disk.

[AdamLepkowski (Adam Łepkowski)]

I thought it works a bit differently but thanks a lot for explanation.