Hello guys,
I was trying to configure a multi node cluster using cloud-on-k8s but I'm unable to achieve that. Basically I created a NFS server as a pod and created two seperate PVs to mount the same end-point so that my 2 node cluster can create 2 PVcs to lock PVs. But it seems I'm getting this error
{"level":"info","ts":1562096740.5542755,"logger":"process-manager","msg":"Update process state","action":"initialization","id":"es","state":"failed","pid":0}
{"level":"info","ts":1562096740.554345,"logger":"process-manager","msg":"Starting..."}
{"level":"info","ts":1562096740.5548103,"logger":"process-manager","msg":"Update process state","action":"start","id":"es","state":"started","pid":18}
{"level":"info","ts":1562096740.554846,"logger":"process-manager","msg":"Started"}
{"level":"info","ts":1562096740.55715,"logger":"keystore-updater","msg":"Waiting for Elasticsearch to be ready"}
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2019-07-02T19:45:41,922+0000", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "ad-tools-cluster", "node.name": "ad-tools-cluster-es-hkj7lr8khw", "message": "uncaught exception in thread [main]" ,
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.1.0.jar:7.1.0]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.1.0.jar:7.1.0]",
"Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?",
"at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:297) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:272) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.0.jar:7.1.0]",
"... 6 more"] }
{"level":"info","ts":1562096741.9779282,"logger":"process-manager","msg":"Update process state","action":"terminate","id":"es","state":"failed","pid":18}
{"level":"info","ts":1562096741.9779992,"logger":"process-manager","msg":"Exit","reason":"process failed"
This is what I see on Kubernetes logs for the second node which is trying to start-up. I tested my NFS volume(by creating a pod that mounts and writes dates) and it does work. I think I'm going wrong with respect to the way PVs are supposed to be configured.