Elasticsearch server is not starting on Ramdisk (imdisk)

Elastic Stack Elasticsearch
1

I am unable to start Elasticsearch when path.data was set to a location on Ramdisk (Imdisk). It throws the following error:

    elasticsearch-7.6.0-windows-x86_64\elasticsearch-7.6.0\bin>.\elasticsearch.bat
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2020-03-04T15:55:49,124][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ALH-MAW7-DT] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[R:\elasticsearch7.6-data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.6.0.jar:7.6.0]
    at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.6.0.jar:7.6.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.6.0.jar:7.6.0]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[R:\elasticsearch7.6-data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
    at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:301) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.node.Node.<init>(Node.java:257) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.0.jar:7.6.0]
    ... 6 more
Caused by: java.io.IOException: failed to obtain lock on R:\elasticsearch7.6-data\nodes\0
    at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:223) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:271) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.node.Node.<init>(Node.java:257) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.0.jar:7.6.0]
    ... 6 more
Caused by: java.nio.file.FileSystemException: R:\elasticsearch7.6-data\nodes\0: Incorrect function.

    at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92) ~[?:?]
    at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103) ~[?:?]
    at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108) ~[?:?]
    at sun.nio.fs.WindowsLinkSupport.getFinalPath(WindowsLinkSupport.java:82) ~[?:?]
    at sun.nio.fs.WindowsFileStore.create(WindowsFileStore.java:87) ~[?:?]
    at sun.nio.fs.WindowsFileSystemProvider.getFileStore(WindowsFileSystemProvider.java:484) ~[?:?]
    at java.nio.file.Files.getFileStore(Files.java:1491) ~[?:?]
    at org.elasticsearch.env.Environment.getFileStore(Environment.java:327) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.env.NodeEnvironment$NodePath.<init>(NodeEnvironment.java:112) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:217) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:271) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.node.Node.<init>(Node.java:257) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.0.jar:7.6.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.0.jar:7.6.0]
    ... 6 more
uncaught exception in thread [main]
java.lang.IllegalStateException: failed to obtain node locks, tried [[R:\elasticsearch7.6-data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
Likely root cause: java.nio.file.FileSystemException: R:\elasticsearch7.6-data\nodes\0: Incorrect function.

    at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
    at java.base/sun.nio.fs.WindowsLinkSupport.getFinalPath(WindowsLinkSupport.java:82)
    at java.base/sun.nio.fs.WindowsFileStore.create(WindowsFileStore.java:87)
    at java.base/sun.nio.fs.WindowsFileSystemProvider.getFileStore(WindowsFileSystemProvider.java:484)
    at java.base/java.nio.file.Files.getFileStore(Files.java:1491)
    at org.elasticsearch.env.Environment.getFileStore(Environment.java:327)
    at org.elasticsearch.env.NodeEnvironment$NodePath.<init>(NodeEnvironment.java:112)
    at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:217)
    at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:271)
    at org.elasticsearch.node.Node.<init>(Node.java:277)
    at org.elasticsearch.node.Node.<init>(Node.java:257)
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
    at org.elasticsearch.cli.Command.main(Command.java:90)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)

Are there any workarounds for this issue? I never had this issue on elasticsearch-5.5.0.

2

Is the mount writable?

3

Yes. I see below lock file was created:
R:\elasticsearch7.6-data\nodes\0\node.lock

I would like to add my investigation so far. Regardless of your java version (8 to 13), we get below exception while executing the Java statement java.nio.file.Files.getFileStore(path) //where “path” points to my Ramdisk drive named R:\ .

java.nio.file.FileSystemException: R:\: Incorrect function.

       at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
       at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
       at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
       at java.base/sun.nio.fs.WindowsFileStore.create(WindowsFileStore.java:94)
       at java.base/sun.nio.fs.WindowsFileSystemProvider.getFileStore(WindowsFileSystemProvider.java:487)
       at java.base/java.nio.file.Files.getFileStore(Files.java:1490)

Hence, any Elasticsearch server (version 7.1 to latest) would throw the above exception if booted on Ramdisk data folder. The problem is with the file https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/env/Environment.java#L322 .
The function getFileStore(Path) is making call to Java NIO API Files.getFileStore(path) to fetch FileStore object.

In Elasticsearch versions 5.5, we dont see these exception because it was using a clever hack to get FileStore object. See https://github.com/elastic/elasticsearch/blob/v5.5.0/core/src/main/java/org/elasticsearch/env/Environment.java#L365 .

4

In order to overcome my situation, I made following changes to code : https://github.com/madhukar-git/elasticsearch/commit/da3d2aa9c1aa26ff780cf0e8a06e35c26078a9ea

But I am still interested if there is a better way to work with RAMDISK.

5

Hi @warkolm , any thoughts?

6

Nope sorry, this is outside the scope of my area.

I'd raise an issue on GitHub, especially as you made that code change.

7

Ok sure. I will try raising an issue at Github.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.