Error after restarting the elasticsearch pod managed by ECK

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I am currently using ECK version 1.2 and Elasticsearch version 7.10.0 managed by ECK. I have restarted one of the elasticsearch pods and I am getting the below errors after pod has restarted.

kubectl logs -n eo elastic-es-ingest-data-0
{"type": "server", "timestamp": "2020-11-19T06:24:33,404Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "elastic", "node.name": "elastic-es-ingest-data-0", "message": "uncaught exception in thread [main]",
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.10.0.jar:7.10.0]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.10.0.jar:7.10.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.10.0.jar:7.10.0]",
"Caused by: java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)",
"at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:72) ~[elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:306) ~[elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:253) ~[elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.bootstrap.Security.configure(Security.java:121) ~[elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:222) ~[elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.10.0.jar:7.10.0]",
"... 6 more",
"Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data",
"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
"at sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:312) ~[?:?]",
"at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:394) ~[elasticsearch-7.10.0.jar:7.10.0]",
uncaught exception in thread [main]
"at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:306) ~[elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:253) ~[elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.bootstrap.Security.configure(Security.java:121) ~[elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:222) ~[elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.10.0.jar:7.10.0]",
"... 6 more"] }

I am currently doing all this for testing purposes, once everything is working fine then I will implement this in production.
So I am desperately looking for a solution for this issue
Answer would be really appreciated.

Thank you.

2

Was the cluster working correctly until you restarted the pod? What is the volume configuration you are using? Please provide your Elasticsearch manifest and information about your Kubernetes cluster -- such as the version, flavour (GKE, EKS, On-prem etc.) and whether you have pod security policies configured.

3

Hi charith,

  • The cluster is configured currectly and it is working fine before restart.

  • Kubernetes version - 1.13

  • Volume I am using mapr fs, (pvc)

  • On-prem

  • No pod security polocies configured.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
spec:
auth: {}
http:
service:
metadata:
creationTimestamp: null
spec:
ports:
- name: https
port: 9200
protocol: TCP
targetPort: 9200
type: LoadBalancer
tls:
certificate: {}
nodeSets:

  • config:
    node.data: false
    node.ingest: false
    node.master: true
    count: 3
    name: master
    podTemplate:
    metadata:
    creationTimestamp: null
    spec:
    containers:
    - name: elasticsearch
    resources: {}
    volumeMounts:
    - mountPath: /es-snaps
    name: es-snap-repo-vol
    initContainers:
    - command:
    - sh
    - -c
    - sysctl -w vm.max_map_count=262144
    name: sysctl
    resources: {}
    securityContext:
    privileged: true
    volumes:
    - name: es-snap-repo-vol
    persistentVolumeClaim:
    claimName: es-snap-repo-claim
    volumeClaimTemplates:
    • metadata:
      creationTimestamp: null
      name: elasticsearch-data
      spec:
      accessModes:
      • ReadWriteOnce
        resources:
        requests:
        storage: 10Gi
        storageClassName: sc-eo
        status: {}
  • config:
    node.data: true
    node.ingest: true
    node.master: false
    path.repo:
    • /es-snaps
      count: 3
      name: ingest-data
      podTemplate:
      metadata:
      creationTimestamp: null
      spec:
      affinity:
      podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
      labelSelector:
      matchLabels:
      elasticsearch.k8s.elastic.co/cluster-name: quickstart
      topologyKey: kubernetes.io/hostname
      weight: 100
      containers:
      • name: elasticsearch
        resources: {}
        volumeMounts:
        • mountPath: /es-snaps
          name: es-snap-repo-vol
          initContainers:
      • command:
        • sh
        • -c
        • sysctl -w vm.max_map_count=262144
          name: sysctl
          resources: {}
          securityContext:
          privileged: true
          volumes:
      • name: es-snap-repo-vol
        persistentVolumeClaim:
        claimName: es-snap-repo-claim
        volumeClaimTemplates:
    • metadata:
      creationTimestamp: null
      name: elasticsearch-data
      spec:
      accessModes:
      • ReadWriteOnce
        resources:
        requests:
        storage: 50Gi
        storageClassName: sc-eo
        status: {}
        transport:
        service:
        metadata:
        creationTimestamp: null
        spec: {}
        updateStrategy:
        changeBudget: {}
        version: 7.10.0
4

Sorry it is not coming nicely the Yaml file

5

You can use Markdown-style triple backticks (```) to enclose the YAML. It is difficult to read in its current form.

Also, please include the output from kubectl describe elastic-es-ingest-data-0 -n eo as well.

6

Yes I am trying but I wont be able to post the yaml in correct format.
While describing the pod I am getting
kubectl describe -n eo elastic-es-ingest-data-0
error: the server doesn't have a resource type "elastic-es-ingest-data-0"
because it havent even started.

7

Sorry I made a mistake with the command, it should be: kubectl describe pod elastic-es-ingest-data-0 -n eo.

Not sure I understand why you can't format the YAML though. It really helps to see YAML in the correct format because spacing is significant.