Error with Elasticsearch lab environment

[Keith_Sanks]

I'm currently enrolled and taking the On-Demand Elasticsearch Engineering 1 course, and I've run into an issue, the course lab informs me to make edits to the elasticsearch.yml, after making the edit to the elasticsearch the lab then says to restart elasticsearch, unfortunately it fails giving the error information below. Since the lab environment is provided by Elastic, I'm really not sure what the issue is, I also don't want to waste the time allotted for the lab to troubleshoot issues that have nothing to do with the course.

"[2020-09-09T15:31:58,222][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/home/elastic/elasticsearch-7.3.1/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.3.1.jar:7.3.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.3.1.jar:7.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.3.1.jar:7.3.1]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/home/elastic/elasticsearch-7.3.1/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:299) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.node.Node.(Node.java:258) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.1.jar:7.3.1]
... 6 more"

Very respectfully,
Keith

[Mark_Laney]

"Failed to obtain node locks" probably means you are trying to run more than one node and the server is sized for just one. Try running this command to find the node's process id.
ps aux | grep elasticsearch
Then find the pid and do:
kill <pid>
(replace with the process id returned by ps) After you kill the node, then try starting it back up again.

[Keith_Sanks]

Not having root access I'm unable to kill processes within an Elastic lab environment.

[pmusa (Pablo Musa)]

Hi Keith,

Not having root access I'm unable to kill processes within an Elastic lab environment.

I doubt that is the case. Elasticsearch cannot be started by the root user and I don't think any other user would start Elasticsearch besides the elastic user. Did you try both commands suggested by Mark? What was the output?

[Keith_Sanks]

pmusa, I'm not sure how familiar you might be with Linux but I was able to do ps -aux | grep elastic and that worked fine, but I was unable to kill the process as it said permission denied.

[Mark_Laney]

When I do the ps the output starts with elastic 55 11.0 then a bunch more. I did kill 55 and reran the ps and pid 55/elasticsearch is no longer running. Does yours behave differently, Keith?

[Keith_Sanks]

It seems to behave differently, I'm not sure if this has to do with the fact that this is an instance of the On-Demand elasticsearchs lab environment running in an AWS for the Elasticsearch Engineering 1 Course. I can't say I know much about the linux version that Elastic is running on, but I can say that the more I try and troubleshoot through this issue the more time I lose for what the lab time is intended for.

[Mark_Laney]

Hi Keith - before the ps and kill check that you run ssh server1. The prompt should say "[elastic@server1]" after that.