Elasticsearch 6.0: failed to obtain node locks

[lanlin (兰林)]

failed to start multi instance on one server after upgrade from 5.x to 6.0.

system: centos 7.4
elasticsearch: 6.0.1

i use the systemd to start elasticsearch, one master node and two data node. it works fine under 5.x version, but failed in 6.0.

Here's the logs

Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: [2017-12-11T10:59:05,646][INFO ][o.e.n.Node               ] [] initializing ...
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: [2017-12-11T10:59:05,687][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/ela
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:134) ~[elasticsearch-6.0.1.jar:6.0.1]
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) ~[elasticsearch-6.0.1.jar:6.0.1]
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) ~[elasticsearch-6.0.1.jar:6.0.1]
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-6.0.1.jar:6.0.1]
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-6.0.1.jar:6.0.1]
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.0.1.jar:6.0.1]
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.0.1.jar:6.0.1]
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch/elasticsearch]] with loc
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:244) ~[elasticsearch-6.0.1.jar:6.0.1]
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: at org.elasticsearch.node.Node.<init>(Node.java:263) ~[elasticsearch-6.0.1.jar:6.0.1]
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-6.0.1.jar:6.0.1]
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.0.1.jar:6.0.1]
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.0.1.jar:6.0.1]
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) ~[elasticsearch-6.0.1.jar:6.0.1]
Dec 11 10:59:05 caf3c89c5314 elasticsearch[1088]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) ~[elasticsearch-6.0.1.jar:6.0.1]

it seems that the "path.data" config not working. All three noses still using /var/lib/elasticsearch/elasticsearch as the data path.

how to resolve this?

[jasontedor]

Would you please share the contents of:

• /etc/elasticsearch/elasticsearch.yml
• /etc/default/elasticsearch
• the initial log line that shows JVM arguments

[lanlin (兰林)]

Sorry, late for my reply. I'v aready sovled this problem.

After install elasticsearch with RPM in centos 7, the default elasticsearch.service can only start one instance. So i create another service files like below to start multiple instances.

All services has it's own yml, jvm.options, log4j2.properties and instance.service file.

As you see, stam0 is one of the service files.

After i open the "elasticsearch-env" bin file, i found the reason.

path "/usr/share/elasticsearch/bin/elasticsearch-env"

No matter how you setting your configs, the "elasticsearch-env" aways load the default
config from "/etc/sysconfig/elasticsearch", your custom configs been override.

Finally, i remove /etc/sysconfig/elasticsearch and all my instances started.

Thanks anyway.

[jasontedor]

You should not edit the scripts. Instead, only put environment variables in /etc/sysconfig/elasticsearch that can be shared across instances. For environment variable overrides otherwise, use instance-specific unit overrides files.

[lanlin (兰林)]

Well, you may misunderstand me. I didn't edit the scripts, i just run a simple remove command

sudo rm -rf /etc/sysconfig/elasticsearch

After that the "elasticsearch-env" can't find the default configs, so my own configs not been override by the default configs.

If i don't do that, only one instance can be started, not all.

[jasontedor]

I understood but I misspoke. I meant you should not remove the /etc/sysconfig/elasticsearch script (note: it's a script for sourcing environment variables in the elasticsearch-env script), in the future the check for existence will be removed and your process will fail to start without it. Instead, leave it present with shared environment variables (e.g., remove ES_PATH_CONF).

[lanlin (兰林)]

Ok, thanks. Next time i will try the way that you suggested.

[jasontedor]

You're welcome, and sorry for the misspeak, I am sure that was confusing.