Cannot deploy eck elasticsearch cluster after successful operator install

[levon.tumanyan (Levon)]

Hi All,

As mentioned I am trying to deploy eck elasticsearch through a manifest install. No helm used.

It is failing on the first init container with the following message:

bash: /mnt/elastic-internal/scripts/prepare-fs.sh: /usr/bin/env: bad interpreter: Permission denied

Here is the elasticsearch yml I am using. I think the issue is that the prepare-fs.sh script is owned by root user and elasticsearch group. I tried playing around with the settings for : "runAsUser: 1000, runAsGroup: 0, fsGroup: 1000"
but no success.

---
# Source: eck-elasticsearch/templates/elasticsearch.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elastic-eck-elasticsearch
  namespace: etss-elastic
  #labels:
  #  app.kubernetes.io/name: eck-elasticsearch
  #  app.kubernetes.io/instance: elastic
  #  annotations:
  #    eck.k8s.elastic.co/license: enterprise
spec:
  version: 8.6.0
  image: <image_url>/elasticsearch_8.6.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
    podTemplate:
        spec:
          securityContext:
            runAsUser: 1000
            runAsGroup: 0
            fsGroup: 1000
          initContainers:
          - name: elastic-internal-init-filesystem
            securityContext:
            #  runAsUser: 0
            #  runAsGroup: 0
            command: ['cat', '/dev/null']
          - name: elastic-internal-suspend
            command: ['tail', '-f', '/dev/null']
          containers:
          - name: elasticsearch
            resources:
              limits:
                memory: 2Gi
              requests:
                cpu: 1
          tolerations:
          - effect: NoSchedule
            key: cloud.rbc.com/agentpool
            value: spark
            operator: Equal

[levon.tumanyan (Levon)]

The original file I used before changing to debug the issue is:

---
# Source: eck-elasticsearch/templates/elasticsearch.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elastic-eck-elasticsearch
  namespace: etss-elastic
  #labels:
  #  app.kubernetes.io/name: eck-elasticsearch
  #  app.kubernetes.io/instance: elastic
  #  annotations:
  #    eck.k8s.elastic.co/license: enterprise
spec:
  version: 8.6.0
  image: <image_url>/elasticsearch_8.6.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
    podTemplate:
        spec:
          containers:
          - name: elasticsearch
            resources:
              limits:
                memory: 2Gi
              requests:
                cpu: 1
          tolerations:
          - effect: NoSchedule
            key: cloud.rbc.com/agentpool
            value: spark
            operator: Equal

[levon.tumanyan (Levon)]

btw this might be useful as well:

elasticsearch@elastic-eck-elasticsearch-es-default-0:/mnt/elastic-internal/scripts$ pwd
/mnt/elastic-internal/scripts
elasticsearch@elastic-eck-elasticsearch-es-default-0:/mnt/elastic-internal/scripts$ ls -la
total 12
drwxrwsrwx 3 root elasticsearch 4096 Jan 24 01:26 .
drwxr-xr-x 8 root root          4096 Jan 24 01:27 ..
drwxr-sr-x 2 root elasticsearch 4096 Jan 24 01:26 ..2023_01_24_01_26_56.578443526
lrwxrwxrwx 1 root elasticsearch   31 Jan 24 01:26 ..data -> ..2023_01_24_01_26_56.578443526
lrwxrwxrwx 1 root elasticsearch   30 Jan 24 01:26 pre-stop-hook-script.sh -> ..data/pre-stop-hook-script.sh
lrwxrwxrwx 1 root elasticsearch   20 Jan 24 01:26 prepare-fs.sh -> ..data/prepare-fs.sh
lrwxrwxrwx 1 root elasticsearch   32 Jan 24 01:26 readiness-probe-script.sh -> ..data/readiness-probe-script.sh
lrwxrwxrwx 1 root elasticsearch   17 Jan 24 01:26 suspend.sh -> ..data/suspend.sh
lrwxrwxrwx 1 root elasticsearch   25 Jan 24 01:26 suspended_pods.txt -> ..data/suspended_pods.txt