Why NFS is to be avoided for data directories

Elastic Stack Elasticsearch
1

ES: 7.5

I have seen the recommendations in official documentation to 'avoid NFS' when using them as data directories for elasticsearch. I have also seen almost everywhere in the forums to 'avoid NFS' but couldn't find a proper explanation.

  1. Is it due to the fact that NFS is exported as a file system, and not a block device? Since there would be higher number of concurrent writes and reads, NFS would have protocol level bottlenecks.

  2. Is it due to network latency related issues of any network related storage (against local storage).

Is it only the (1) reason, if that so, can we use a network bloack storage like ceph or SAN.

Thanks

2

Main reason to me is the latency.

1 Like
3

Yes, latency is a big factor. Another is correctness: Elasticsearch expects the filesystem under the data path to act like a local filesystem, and uses some fairly advanced features that are typically not well supported by nonlocal storage. For instance it needs locking and atomic file creation to work right; it's notoriously tricky to get NFS to do these things correctly.

Distributed storage like Ceph and GlusterFS is something I'd avoid. These technologies are still maturing IMO and have been linked to lost or corrupt data in the recent past. You don't need distributed storage since Elasticsearch handles the distributed side of things for you.

SANs work ok where performance is less important (e.g. the cold tier). I haven't heard of as many correctness issues with SANs as with your other suggestions.

3 Likes
4

As far as I recall it is both. There used to be issues around NFS causing problems due to not behaving and offering the same mechanisms as a block store, but that may have been addressed in more recent versions. Performance is a however still a major consideration as Elasticsearch tends to perform a lot of small random reads and writes rather that large consecutive operations. I do not recall seeing ceph used but many users have clusters backed by SAN. SAN performance can vary a lot so make sure you test properly.

1 Like
5

Hi Guys,

Thanks for the quick help.
Our infrastructure team wants to use already available NFS storage for this task. I'll let them know that this unsuitable for elasticsearch.

1 Like
6

.... and while you are avoiding NFS, also avoid spinning hard drives - the pain is real.

SSD is the best starting point .

7

Spinning disks are certainly a poor choice for nodes that see heavy indexing, but they can work well for nodes in the warm and cold tiers that see very little write traffic.

1 Like
8

Your right, for warm or cold tiers spinning drives might be perfectly fine.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.