Can't configure a multi-node cluster using Elastic-Kubernetes

Hello guys,
I was trying to configure a multi node cluster using cloud-on-k8s but I'm unable to achieve that. Basically I created a NFS server as a pod and created two seperate PVs to mount the same end-point so that my 2 node cluster can create 2 PVcs to lock PVs. But it seems I'm getting this error
{"level":"info","ts":1562096740.5542755,"logger":"process-manager","msg":"Update process state","action":"initialization","id":"es","state":"failed","pid":0}

{"level":"info","ts":1562096740.554345,"logger":"process-manager","msg":"Starting..."}

{"level":"info","ts":1562096740.5548103,"logger":"process-manager","msg":"Update process state","action":"start","id":"es","state":"started","pid":18}

{"level":"info","ts":1562096740.554846,"logger":"process-manager","msg":"Started"}

{"level":"info","ts":1562096740.55715,"logger":"keystore-updater","msg":"Waiting for Elasticsearch to be ready"}

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

{"type": "server", "timestamp": "2019-07-02T19:45:41,922+0000", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "ad-tools-cluster", "node.name": "ad-tools-cluster-es-hkj7lr8khw", "message": "uncaught exception in thread [main]" ,

"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?",

"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.1.0.jar:7.1.0]",

"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.1.0.jar:7.1.0]",

"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.1.0.jar:7.1.0]",

"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.1.0.jar:7.1.0]",

"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.1.0.jar:7.1.0]",

"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.1.0.jar:7.1.0]",

"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.1.0.jar:7.1.0]",

"Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?",

"at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:297) ~[elasticsearch-7.1.0.jar:7.1.0]",

"at org.elasticsearch.node.Node.<init>(Node.java:272) ~[elasticsearch-7.1.0.jar:7.1.0]",

"at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.0.jar:7.1.0]",

"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]",

"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]",

"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.0.jar:7.1.0]",

"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.0.jar:7.1.0]",

"... 6 more"] }

{"level":"info","ts":1562096741.9779282,"logger":"process-manager","msg":"Update process state","action":"terminate","id":"es","state":"failed","pid":18}

{"level":"info","ts":1562096741.9779992,"logger":"process-manager","msg":"Exit","reason":"process failed"

This is what I see on Kubernetes logs for the second node which is trying to start-up. I tested my NFS volume(by creating a pod that mounts and writes dates) and it does work. I think I'm going wrong with respect to the way PVs are supposed to be configured.

Please don't post images of text as they are hardly readable and not searchable.

Instead paste the text and format it with </> icon. Check the preview window.

few things:

  • do not share the same dir for all elasticsearch instances. Each elasticsearch should write in its own directory. That's why you are seeing this message.
  • prefer using local disks always instead of network drive as it tends to be really slow

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.