Unable to start non-data nodes Elasticsearch after upgrading from 6.42 to 6.6.1

Elastic Stack Elasticsearch
1

I upgraded my cluster based on the instructions from the link

After upgrading none of the non-data nodes started.
All of them had the below error
[2019-02-20T22:57:35,922][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/etc/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

I even tried setting node.max_local_storage_nodes the value to 5 and started and still the error persisted.

2

How many nodes are running locally?

3

only 1

4

Then what is the problem if the node is already running? Why do you want to start another one?

5

I had installed the Elasticsearch via yum install so i have only one Elasticsearch running. This error is appearing when I do a service elasticsearch restart

6

Pasting the full stack trace

[2019-02-21T03:58:35,885][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to create node environment
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.6.1.jar:6.6.1]
Caused by: java.lang.IllegalStateException: Failed to create node environment
	at org.elasticsearch.node.Node.<init>(Node.java:298) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]
	... 6 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/data/nodes/0
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:?]
	at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_181]
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_181]
	at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_181]
	at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:270) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:203) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.node.Node.<init>(Node.java:295) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]
	... 6 more
^C
[node1:/home/users/usr1]# ps aux |grep java
root     31871  0.0  0.0 112676   728 pts/0    R+   03:58   0:00 grep --color=auto java
[node1:/home/users/usr1]#
7

The error here does not seem to be the same as you mentioned initially:

Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/data/nodes/0

Most likely a permission issue?

8

Thank you @dadoonet

The permission was

drwxr-sr-x 2 root elasticsearch 6 Feb 21 01:08 nodes

updating to elasticsearch:elasticsearch started the node.

But why did change happen during upgrade and that too only in non-data nodes. Is there any special thing that I need to take care while upgrading.

9

I don't know. May be you ran something as root instead of elasticsearch at some point?

10

@dadoonet but the steps to upgrade was run via script it was same on all the nodes including the data nodes. The data nodes are fine post upgrade.

I am just trying to see that if I need to make any specific preparation for my prod setup.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.