Elasticsearch master and client service/pod not coming up

Elastic Stack Elasticsearch
1

I'm running Elasticsearch cluster in openshift environment, where i recently patched and rebooted the underlaying nodes, after reboot master and client pods are not coming up and error seems to be related to mount, but don't see any issue with that since all required volumes are mounted and permissions look good.

Does this version has any dependency with OS version(though its running as pod and there is no change in it) or any other parameter causing this issue.

Openshift version: 3.11
Elasticsearch version: 7.4.2
OS Version: centos 7.9
Docker version: 1.13.1

Error:

wget: server returned error: HTTP/1.1 404 Not Found
Node Zone is dynamically set to us-east4-b (gcp)

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2021-02-01T05:59:57,340][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-0] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.0.jar:7.4.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.0.jar:7.4.0]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:299) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.node.Node.<init>(Node.java:275) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.node.Node.<init>(Node.java:255) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.0.jar:7.4.0]
	... 6 more
Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/0
	at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:269) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.node.Node.<init>(Node.java:275) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.node.Node.<init>(Node.java:255) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.0.jar:7.4.0]
	... 6 more
Caused by: java.io.IOException: Mount point not found
	at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:105) ~[?:?]
	at sun.nio.fs.UnixFileStore.<init>(UnixFileStore.java:67) ~[?:?]
	at sun.nio.fs.LinuxFileStore.<init>(LinuxFileStore.java:49) ~[?:?]
	at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:51) ~[?:?]
	at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:39) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.getFileStore(UnixFileSystemProvider.java:369) ~[?:?]
	at java.nio.file.Files.getFileStore(Files.java:1487) ~[?:?]
	at org.elasticsearch.env.Environment.getFileStore(Environment.java:327) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.env.NodeEnvironment$NodePath.<init>(NodeEnvironment.java:110) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:215) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:269) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.node.Node.<init>(Node.java:275) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.node.Node.<init>(Node.java:255) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.0.jar:7.4.0]
	... 6 more
2

The code that is failing is in the JDK:

My best guess is that /proc/mounts is being manipulated by your environment so that it does not correspond to the actual mount points. That's only a guess, sorry, I don't know what else Openshift might be doing to get in the way of the JDK here.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.