Elasticsearch dead but subsys locked

[ItsMee (Mohan)]

Dear All,

I have tried to restart the elasticsearch but getting 'elasticsearch dead but subsys locked' after start. I have removed the lock the file(elasticsearch) under /var/lock/subsys/ multiple times but no luck.

Could someone please help me here?

[warkolm (Mark Walkom)]

Welcome to our community!

Can you please post your Elasticsearch log. What version are you running?

[ItsMee (Mohan)]

Hi Mark,
Version is 5.6.

[2020-12-02T10:31:55,667][INFO ][o.e.n.Node ] [elasticsearch-node-client-8] initializing ...
[2020-12-02T10:31:55,714][ERROR][o.e.b.Bootstrap ] Exception
java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch/elk_prd_5]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:261) ~[elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.node.Node.(Node.java:265) ~[elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:233) ~[elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) [elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) [elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) [elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) [elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.6.11.jar:5.6.11]
[2020-12-02T10:31:55,723][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-node-client-8] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch/elk_prd_5]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.11.jar:5.6.11]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.11.jar:5.6.11]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch/elk_prd_5]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

[dadoonet (David Pilato)]

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.

Anyway, here the problem is that one node already started and was killed without removing a lock file.
The lock is somewhere in /var/lib/elasticsearch/elk_prd_5.

Note that this version is a way too old to be supported. You should really upgrade to 7.10.

[ItsMee (Mohan)]

Hi David,

Thanks for your reply. I am getting into same situation even after deleting lock.

[dadoonet (David Pilato)]

What exactly did you remove?
What is the output of:

ls -la /var/lib/elasticsearch/elk_prd_5