Not able to start elastic Service

[pnrao.83 (Narasimha Rao Parasa)]

Hi Team,

i am trying to configure and use elastic search and Kibana. But i was stuck at starting the elastic service in my Oracle Linux Machine. Please find the below logs and yaml file and help:
cat /var/log/elasticsearch/elasticsearch.log
[2021-02-05T08:12:55,780][INFO ][o.e.n.Node ] [inoclams2] version[7.10.2], pid[13585], build[default/rpm/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/4.18.0-147.0.3.el8_1.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-02-05T08:12:55,785][INFO ][o.e.n.Node ] [inoclams2] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-02-05T08:12:55,786][INFO ][o.e.n.Node ] [inoclams2] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -[2021-02-05T08:13:00,387][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [lang-expression]
[2021-02-05T08:13:00,387][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [lang-mustache]
[search-business-rules]
[2021-02-05T08:13:00,391][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [searchable-snapshots]
[2021-02-05T08:13:00,392][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [spatial]
[2021-02-05T08:13:00,392][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [systemd]
[2021-02-05T08:13:00,392][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [transform]
[2021-02-05T08:13:00,393][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [transport-netty4]
[2021-02-05T08:13:00,393][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [unsigned-long]
[2021-02-05T08:13:00,393][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [vectors]
[2021-02-05T08:13:00,394][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [wildcard]
[2021-02-05T08:13:00,394][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-analytics]
[2021-02-05T08:13:00,394][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-async]
[2021-02-05T08:13:00,395][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-async-search]
[2021-02-05T08:13:00,395][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-autoscaling]
[2021-02-05T08:13:00,395][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-ccr]
[2021-02-05T08:13:00,396][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-core]
[2021-02-05T08:13:00,396][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-data-streams]
[2021-02-05T08:13:00,397][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-deprecation]
[2021-02-05T08:13:00,397][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-enrich]
[2021-02-05T08:13:00,397][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-eql]
[2021-02-05T08:13:00,398][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-graph]
[2021-02-05T08:13:00,398][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-identity-provider]
[2021-02-05T08:13:00,399][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-ilm]
[2021-02-05T08:13:00,399][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-logstash]
[2021-02-05T08:13:00,399][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-ml]
[2021-02-05T08:13:00,400][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-monitoring]
[2021-02-05T08:13:00,400][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-ql]
[2021-02-05T08:13:00,401][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-rollup]
[2021-02-05T08:13:00,401][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-security]
[2021-02-05T08:13:00,401][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-sql]
[2021-02-05T08:13:00,402][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-stack]
[2021-02-05T08:13:00,402][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-voting-only-node]
[2021-02-05T08:13:00,402][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-watcher]
[2021-02-05T08:13:00,403][INFO ][o.e.p.PluginsService ] [inoclams2] no plugins loaded
[2021-02-05T08:13:00,472][ERROR][o.e.b.Bootstrap ] [inoclams2] Exception
java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:302) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.node.Node.(Node.java:362) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.node.Node.(Node.java:289) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.10.2.jar:7.10.2]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.10.2.jar:7.10.2]
Caused by: java.io.IOException: failed to obtain lock on /var/lib/elasticsearch/nodes/0
obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.10.2.jar:7.10.2]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.10.2.jar:7.10.2]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:302) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.node.Node.(Node.java:362) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.node.Node.(Node.java:289) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.10.2.jar:7.10.2]
... 6 more
Caused by: java.io.IOException: failed to obtain lock on /var/lib/elasticsearch/nodes/0
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:224) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:272) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.node.Node.(Node.java:362) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.node.Node.(Node.java:289) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.10.2.jar:7.10.2]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /var/lib/elasticsearch/nodes/0/node.lock
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:181) ~[?:?]
at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]
at java.nio.channels.FileChannel.open(FileChannel.java:345) ~[?:?]
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:217) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:272) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.node.Node.(Node.java:362) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.node.Node.(Node.java:289) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.10.2.jar:7.10.2]
... 6 more

cat /etc/elasticsearch/elasticsearch.yml

cluster.name lab

node.name lab1

path.data /var/lib/elasticsearch

path.logs /var/log/elasticsearch

bootstrap.memory_lock true

network.host 0.0.0.0

http.port 9200

discovery.type 'single-node'

indices.query.bool.max_clause_count 8192

search.max_buckets 250000

action.destructive_requires_name 'true'

reindex.remote.whitelist ':'

xpack.monitoring.enabled 'true'

xpack.monitoring.collection.enabled 'true'

xpack.monitoring.collection.interval 30s

xpack.security.enabled 'true'

xpack security.audit.enabled 'false'

node.ml 'false'

xpack.ml.enabled 'false'

xpack.watcher.enabled 'false'

xpack.ilm.enabled 'true'

xpack.sql.enabled 'true'

ll /var/lib/elasticsearch/
total 0
drwxr-sr-x. 3 elasticsearch elasticsearch 15 Feb 4 02:53 nodes

[keyofstarrysky973]

Hey I found the following thread : Unable to start elasticsearch as service - #5 by Kajol_Nimesh

It seems that you have a permission issue for the elastic directory /var/lib/elasticsearch :

Caused by: java.io.IOException: failed to obtain lock on /var/lib/elasticsearch/nodes/0

The reported solution is to give root permission to that directory : sudo chown -R elasticsearch:elasticsearch /var/lib/elasticsearch

Whether or not this is best practice from a security standpoint is unkwon to me. Try this solution, and pls tell if it worked for you.

[pnrao.83 (Narasimha Rao Parasa)]

Nope, it dint worked out... i have changed the permission and tried again.. please find the below out put again:
[root@inoclams2 ~]# ls -la /var/lib/elasticsearch/
total 4
drwxr-s---. 3 elasticsearch elasticsearch 19 Jan 13 00:55 .
drwxr-xr-x. 45 root root 4096 Feb 5 08:03 ..
drwxr-sr-x. 3 elasticsearch elasticsearch 15 Feb 4 02:53 nodes
[root@inoclams2 ~]# ls -ld /var/lib/elasticsearch/
drwxr-s---. 3 elasticsearch elasticsearch 19 Jan 13 00:55 /var/lib/elasticsearch/
[root@inoclams2 ~]# chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/
[root@inoclams2 ~]# ls -la /var/lib/elasticsearch/
total 4
drwxr-s---. 3 elasticsearch elasticsearch 19 Jan 13 00:55 .
drwxr-xr-x. 45 root root 4096 Feb 5 08:03 ..
drwxr-sr-x. 3 elasticsearch elasticsearch 15 Feb 4 02:53 nodes
[root@inoclams2 ~]# systemctl start elasticsearch
Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
[root@inoclams2 ~]# cat /var/log/elasticsearch/elasticsearch.log
[2021-02-05T08:12:55,780][INFO ][o.e.n.Node ] [inoclams2] version[7.10.2], pid[13585], build[default/rpm/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/4.18.0-147.0.3.el8_1.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-02-05T08:12:55,785][INFO ][o.e.n.Node ] [inoclams2] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-02-05T08:12:55,786][INFO ][o.e.n.Node ] [inoclams2] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-15173963832289045179, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2021-02-05T08:13:00,383][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [aggs-matrix-stats]
[2021-02-05T08:13:00,402][INFO ][o.e.p.PluginsService ] [inoclams2] loaded module [x-pack-watcher]
[2021-02-05T08:13:00,403][INFO ][o.e.p.PluginsService ] [inoclams2] no plugins loaded
[2021-02-05T08:13:00,472][ERROR][o.e.b.Bootstrap ] [inoclams2] Exception
java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:302) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.node.Node.(Node.java:362) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.node.Node.(Node.java:289) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.10.2.jar:7.10.2]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.10.2.jar:7.10.2]
Caused by: java.io.IOException: failed to obtain lock on /var/lib/elasticsearch/nodes/0
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:224) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:272) ~[elasticsearch-7.10.2.jar:7.10.2]
... 12 more
Caused by: java.nio.file.AccessDeniedException: /var/lib/elasticsearch/nodes/0/node.lock
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:181) ~[?:?]
at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]
at java.nio.channels.FileChannel.open(FileChannel.java:345) ~[?:?]
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:217) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:272) ~[elasticsearch-7.10.2.jar:7.10.2]
... 12 more
[2021-02-05T08:13:00,485][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [inoclams2] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.10.2.jar:7.10.2]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.10.2.jar:7.10.2]

[root@inoclams2 ~]# ll /var/lib/elasticsearch/nodes/0/node.lock
-rw-r--r--. 1 elasticsearch elasticsearch 0 Feb 5 07:04 /var/lib/elasticsearch/nodes/0/node.lock
[root@inoclams2 ~]#

[kavierkoo (Kavier)]

Hi Pnrao,

Please format your codes and errors under preformatted text </> or backticks (```) as it is really hard to read otherwise.

java.lang.IllegalStateException: failed to obtain node locks, 

If this was not due to file permission issue,
Could you check if the process is not being stopped properly (any existing process running even if when

systemctl status elasticsearch

shows inactive?

sudo ps -aux | grep elasticsearch

Hope this can help with the troubleshooting.

[pnrao.83 (Narasimha Rao Parasa)]

[root@inoclams2 ~]# systemctl status elasticsearch'
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/elasticsearch.service.d
' └─elasticsearch.conf Active: failed (Result: exit-code) since Sat 2021-02-06 00:59:23 GMT; 6h ago Docs: https://www.elastic.co Process: 29391 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, statu> Main PID: 29391 (code=exited, status=1/FAILURE)`

Feb 06 00:59:23 inoclams2 systemd-entrypoint[29391]: at org.yaml.snakeyaml.scanner.ScannerImpl.needMoreTokens(ScannerImpl.java>
Feb 06 00:59:23 inoclams2 systemd-entrypoint[29391]: at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:226)
Feb 06 00:59:23 inoclams2 systemd-entrypoint[29391]: at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(Pars>
Feb 06 00:59:23 inoclams2 systemd-entrypoint[29391]: at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
Feb 06 00:59:23 inoclams2 systemd-entrypoint[29391]: at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:168)
Feb 06 00:59:23 inoclams2 systemd-entrypoint[29391]: at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.>
Feb 06 00:59:23 inoclams2 systemd-entrypoint[29391]: ... 14 more
Feb 06 00:59:23 inoclams2 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILUR'E
Feb 06 00:59:23 inoclams2 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Feb 06 00:59:23 inoclams2 systemd[1]: Failed to start Elasticsearch.

[root@inoclams2 ~]# ps -aux | grep elasticsearch
root 2554 0.0 0.0 12108 1052 pts/0 S+ 07:00 0:00 grep --color=auto elasticsearch

My config File:

node.max_local_storage_nodes: 1
path.data: /var/lib/elasticsearch
path.logas: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 132.145.62.111
http.port: 9200
discovery.type 'single-node'
indices.query.bool.max_clause_count 8192
search.max_buckets 250000
action.destructive_requires_name 'true reindex.remote.whitelist ':'
xpack.monitoring.enabled 'true'
xpack.monitoring.collection.enabled 'true'
xpack.monitoring.collection.interval 30s'
xpack.security.enabled 'true' xpack security.audit.enabled 'false'
node.ml 'false'
xpack.ml.enabled 'false'
xpack.watcher.enabled 'false'
xpack.ilm.enabled 'true'
xpack.sql.enabled 'true'`

[keyofstarrysky973]

If it isn't a permission issue, then it's most likely an issue with running processes. There probably is an elasticsearch instance that has not been properly terminated. Can you try changing the default port number, like for example : http.port: 9201