Elastic stack installation issue with NFS

Elastic Stack Elasticsearch
1

Hi,

I installed a basic self-managed Elastic Stack successfully. But I was trying to change path.data to use a NFS mounted path in the server.
When I try to start the elasticsearch service I get the following error:

java.lang.IllegalStateException: failed to obtain node locks, tried [/app/elastic]; maybe these locations are not writable or multiple nodes were started on the same data path?
...
Caused by: java.io.IOException: failed to obtain lock on /app/elastic
...
Caused by: java.nio.file.AccessDeniedException: /app/elastic/node.lock

The fact is that the file is created, but empty.

[jboss@sdeselklin1 app]$ ls -l elastic/
total 32
-rwxrwx---. 1 elasticsearch elasticsearch 0 25 oct. 11:46 node.lock

I have tried to create a file manually and the nfs is working.
The owner and group for the full path is also elasticsearch.

So I am very confused. Any hint?

2

The file system backing the path.data must as outlined in the documentation behave like a local disk. This is often not the case when you use an NFS monted volume, so I suspect this is your issue. Using NFS storage also often results in very poor performance even when it does work and is correctly configured, so I would recommend against using it.

3

I read the documentation but I don't fully understand what "behave like a local disk" mean. Since the path is mounted in the server, you can interact as a local disk. Do you have any suggestion on mount parameters?

We have tried these two configurations.
isinfsdg:/elastic /app/elastic nfs rw,bg,hard,nointr,tcp,vers=3 0 0
isinfsdg:/elastic /app/elastic nfs defaults 0 0

Regarding the poor performance, I know, we are beginning with the stack and need to test with nfs disk space. Unfortunately I cannot decide right now. If we see very very poor performance we'll think on moving to a different filesystem.

4

I have never used Elasticsearch with NFS so do not even know if it is possible for it to work nor how to configure it it if it is. It is not just about being able to read or write to the file system as file locking, durability guarantees and other low-level operations also need to match that of a local disk. Elasticsearch can be very I/O intensive and can put a lot of strain on the storage. You probably need a good system administrator with extensive NFS experience to get it to work as I have seen a lot of people struggle with it over the years.

5 
java.nio.file.AccessDeniedException

Maybe just a permission issue? Elastic user seems to not have access to the path.

Greetings, Pedro

1 Like
6

Maybe, but the directory and preceding directory has the proper permissions for elasticsearch user and group. In fact, the file is created (but with no bytes)

I just checked the creation time in detail, the error log is at 11:46:13,466. The creation date of the file is 11:46:13.520169000. So I think the problem is that our NFS system is slower that elasticsearch can work with.

I close the topic