Failed to obtain node locks, t multiple nodes were started without increasing

sadeshdrj · August 23, 2019, 6:59am

i have a downloaded and installing Elasticsearch security for the basic version of kibana and elasticsearch.
but i do get the following error, i did follow some threads and changed the chown user from root user to profile user, but still im hanging on this error for days,

following link is the method that im trying, https://www.elastic.co/blog/getting-started-with-elasticsearch-security

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be [2019-08-23T15:45:31,271][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [localhost.localdomain] uncaugh org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, t multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])? at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.3.0.jar:7 at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.3.0.ja at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticse at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.3.0 at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.3.0.jar:7.3.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.3.0.jar:7 at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.3.0.jar:7. Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/home/kibana/elasticsearch-7t increasing [node.max_local_storage_nodes] (was [1])? at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:299) ~[elasticsearch-7.3.0.jar:7 at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.3.0.jar:7.3.0] at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.0.jar:7.3.0] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.0.jar:7.3.0 at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.0.jar:7.3.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.0.jar:7.3.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.0.jar:7 ... 6 more

spinscale · August 23, 2019, 7:06am

Hey,

this usually indicates another Elasticsearch process running. Can you share the output of jps? Which other java processes are running?

Is there a file called node.lock in your data directory? If so, what is the timestamp from that file?

--Alex

sadeshdrj · August 23, 2019, 7:17am

Hi Alex,
thank you for your quick respond,

these are the current java process and elasticsearch threads running,

java
kibana 28284 2.7 33.3 3373228 1291312 pts/0 Tl 15:21 1:25 /home/kibana/elasticsearch-7.3.0-master/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-1100976547446380811 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -Dio.netty.allocator.type=unpooled -XX:MaxDirectMemorySize=536870912 -Des.path.home=/home/kibana/elasticsearch-7.3.0-master -Des.path.conf=/home/kibana/elasticsearch-7.3.0-master/config -Des.distribution.flavor=default -Des.distribution.type=tar -Des.bundled_jdk=true -cp /home/kibana/elasticsearch-7.3.0-master/lib/* org.elasticsearch.bootstrap.Elasticsearch
kibana 28962 0.0 0.0 112708 976 pts/0 R+ 16:12 0:00 grep --color=auto java

elasticsearch

kibana   27977  0.0  0.0 126316  1748 pts/0    T    15:14   0:00 vi config/elasticsearch.yaml
kibana   28284  2.6 33.3 3373228 1291312 pts/0 Tl   15:21   1:25 /home/kibana/elasticsearch-7.3.0-master/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-1100976547446380811 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -Dio.netty.allocator.type=unpooled -XX:MaxDirectMemorySize=536870912 -Des.path.home=/home/kibana/elasticsearch-7.3.0-master -Des.path.conf=/home/kibana/elasticsearch-7.3.0-master/config -Des.distribution.flavor=default -Des.distribution.type=tar -Des.bundled_jdk=true -cp /home/kibana/elasticsearch-7.3.0-master/lib/* org.elasticsearch.bootstrap.Elasticsearch
kibana   28361  0.0  0.1  68820  5532 pts/0    Tl   15:21   0:00 /home/kibana/elasticsearch-7.3.0-master/modules/x-pack-ml/platform/linux-x86_64/bin/controller
kibana   28964  0.0  0.0 112708   984 pts/0    R+   16:15   0:00 grep --color=auto elasticsearch

spinscale · August 23, 2019, 7:26am

this shows, you already have an elasticsearch process running

sadeshdrj · August 23, 2019, 7:41am

i have killed all the elasticsearches that running already and run the command bin/elasticsearch

in the end i get the following error,

[2019-08-23T16:37:50,097][INFO ][o.e.p.PluginsService ] [localhost.localdomain] no plugins loaded [2019-08-23T16:38:01,867][INFO ][o.e.x.s.a.s.FileRolesStore] [localhost.localdomain] parsed [0] roles from file [/home/kibana/elasticsearch-7.3.0-master/config/roles.yml] [2019-08-23T16:38:03,555][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [localhost.localdomain] [controller/29377] [Main.cc@110] controller (64 bit): Version 7.3.0 (Build ff2f774f78ce63) Copyright (c) 2019 Elasticsearch BV [2019-08-23T16:38:04,744][DEBUG][o.e.a.ActionModule ] [localhost.localdomain] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security [2019-08-23T16:38:06,862][INFO ][o.e.d.DiscoveryModule ] [localhost.localdomain] using discovery type [zen] and seed hosts providers [settings] [2019-08-23T16:38:12,111][INFO ][o.e.n.Node ] [localhost.localdomain] initialized [2019-08-23T16:38:12,112][INFO ][o.e.n.Node ] [localhost.localdomain] starting ... [2019-08-23T16:38:12,890][INFO ][o.e.t.TransportService ] [localhost.localdomain] publish_address {192.168.XX.XXX:9300}, bound_addresses {[::]:9300} [2019-08-23T16:38:12,936][INFO ][o.e.b.BootstrapChecks ] [localhost.localdomain] bound or publishing to a non-loopback address, enforcing bootstrap checks ERROR: [3] bootstrap checks failed [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535] [2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] [3]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured [2019-08-23T16:38:13,031][INFO ][o.e.n.Node ] [localhost.localdomain] stopping ... [2019-08-23T16:38:13,101][INFO ][o.e.n.Node ] [localhost.localdomain] stopped [2019-08-23T16:38:13,102][INFO ][o.e.n.Node ] [localhost.localdomain] closing ... [2019-08-23T16:38:13,175][INFO ][o.e.n.Node ] [localhost.localdomain] closed [2019-08-23T16:38:13,181][INFO ][o.e.x.m.p.NativeController] [localhost.localdomain] Native controller process has stopped - no new native processes can be started

ikakavas · August 23, 2019, 8:01am

Please go through https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html, everything in your errors above is discussed and explained there

system · September 20, 2019, 8:01am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.