Elastic search pod CrashLoopBackOff's in multi master kubernetes cluster

Elastic Stack Elasticsearch
1

Below is the yaml of the elastic search deployment that I created for my single node k8s cluster, the same when I run on my multi master (3 master 6 worker) cluster, my pod is in crashloopbackoff status. I am using rook-ceph v0.9 as my storage provisioner. What changes do I need to do so this pod can work in moth multi master as well as single master multi worker clusters ?

Deployment yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: en-sysconfig-es-db
  namespace: default
spec:
  template:
    metadata:
      labels:
        app: en-sysconfig-es-db
    spec:
      containers:
        - name: en-sysconfig-es-db
          image: docker.elastic.co/elasticsearch/elasticsearch:7.4.0
          env:
          - name: discovery.type
            value: single-node
          ports:
            - containerPort: 9200
          volumeMounts:
            - name: en-sysconfig-es-db-persistent-storage
              mountPath: /usr/share/elasticsearch/data
      volumes:
      - name: en-sysconfig-es-db-persistent-storage
        persistentVolumeClaim:
          claimName: en-sysconfig-es-db-claim
---
apiVersion: v1
kind: Service
metadata:
  name: en-sysconfig-es-db-service
  namespace: default
  labels:
    name: en-sysconfig-es-db-service
spec:
  ports:
    - port: 9200
      targetPort: 9200
      protocol: TCP
  selector:
    app: en-sysconfig
  type: ClusterIP

pvc yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: en-sysconfig-es-db-claim
  namespace: default
spec:
  storageClassName:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi

Logs from kubernetes

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2019-10-03T09:25:46,198Z", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "docker-cluster", "node.name": "en-sysconfig-es-db-77d95cc85f-4j86h", "message": "uncaught exception in thread [main]",
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.0.jar:7.4.0]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.0.jar:7.4.0]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.<init>(Node.java:614) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:255) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.0.jar:7.4.0]",
"... 6 more",
"Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]",
"at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]",
"at java.nio.file.Files.createDirectory(Files.java:693) ~[?:?]",
"at java.nio.file.Files.createAndCheckIsDirectory(Files.java:800) ~[?:?]",
"at java.nio.file.Files.createDirectories(Files.java:786) ~[?:?]",
"at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:272) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:209) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:269) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:275) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:255) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.0.jar:7.4.0]",
```
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.