Possibilities of increasing data security in elasticsearch - power failures [?]

Elastic Stack Elasticsearch
1

Hi,

Relatively simple question about the configuration of elasticsearch with respect to specific data security.

Are there any possibilities to configure an elasticsearch cluster so that in case of a power failure there are no additional data losses?

What security models when not all nodes have redundant power supply?

Is there anything we can do?

Of course backup is backup. But I'm looking for a quick solution in elasticsearch.

2

By default Elasticsearch should lose no acknowledged writes when power is lost, assuming your storage system implements fsync() correctly. Lucene in particular has seen some fairly rigorous testing of its safety in the face of power loss.

Be warned that it's relatively common to lose data in a power outage because your storage system does not in fact implement fsync() correctly, but there's not much Elasticsearch can do about that. You can run some tests to try and detect this; as long as your system passes those sorts of tests then Elasticsearch should be fine.

3

Thanks a lot!

I'll test this software...

I've noticed that if there are problems, it's only in the indexes where the documents are also saved.

Are read-only indexes more robust?
If I only read in indexes, is elasticsearch not writing any data to disk and it is resistant to such problems (power problems)?

4

Probably yes, most broken fsync() implementations will still take effect within a few seconds so in practice you only notice the problem on data that was written in the last few seconds before a power outage.

5

I understand.
fsync() is only implemented at the software level or is it also hardware dependent?
SSDs have better support than normal HDDs for fsync()?

6

It's software, but that includes the firmware running on your drives and all sorts of system-dependent configuration. I don't know of any general differences between SSDs and spinning disks. I'm pretty sure you can configure any drive to do it properly, it's just not always the default.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.