Weird manual workaround nedded for APM server and Maps server under ECK on OpenShift

Hello all,

until recently I had Elastic stack running under ECK on OpenShift with no problems. However, after an upgrade to 9.0.0 (and now 9.0.1) APM server and Maps server stopped working, OpenShift emitting a weird error about executable not found in $PATH. So I did some manual digging in ECK-generated manifests and finally I went past the issues by modifying podTemplates for these resources like this:

APM server:

podTemplate:
      spec:
        containers:
          - name: apm-server
            env:
              - name: PATH
                value: "/bin:/usr/bin:/usr/share/apm-server"
            volumeMounts: 
              - name: non-persistent
                mountPath: /usr/share/apm-server/data
                readOnly: false
        volumes:
          - name: non-persistent
            emptyDir: {}

When finally starting the container, there was another error about not being able to create the data directory, hence the emptyDir.

Maps server:

podTemplate:
      spec:
        containers:
          - name: maps
            command:
              - /bin/sh
              - -c
              - "cd /usr/src/app/server/app && node index.js"

So I've got it working. However, I hardly think this should be compulsory, or is it? I didn't find these issues documented anywhere. Is there a better way?

Thank you.

Hi,

I can't reproduce the issue for the APM server. Could you share the original manifest and your OpenShift version?

With regard to Maps I can confirm that the image does not start since 9.0.0:

Warning  Failed          17m (x9 over 18m)     kubelet            Error: container create failed: open executable: Operation not permitted

I'll create a GitHub issue in the ECK repo and try to understand the root cause.

Hello Michael and thank you for your response.

OpenShift version is 4.16.30 and I deployed APM server via Elastic helm chart with the following values.yaml before adding the podTemplate part:

fullnameOverride: "apm-server-poc"
version: 9.0.1
count: 1
config:
  monitoring:
    enabled: false
    cluster_uuid: ZJn6TF45SOuEHCkdPCgVcQ
  http: 
    enabled: true
    port: 5066
    host: 0.0.0.0
http:
  service:
    spec:
      ports:
        - name: http
          port: 8200
          targetPort: 8200
        - name: metrics
          port: 5066
          targetPort: 5066
elasticsearchRef: 
  name: elastic-poc
kibanaRef:
  name: kibana-poc

Regarding the Maps server I got a different error from yours. It was the same error as with the APM server about binary not being found in $PATH.

I also have APM and Maps server on another infrastructure where they run as containers as well (no k8s or OpenShift there, just Docker) and they work without problems.

Interesting, I'm using 4.18.1.

For Maps the node binary is in /usr/bin/node, which is in the PATH variable: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" This variable was not changed between 8.17 and 9.0.0. So I don't understand why you would get an error about the binary not in $PATH.

I created ElasticMapsServer does not start on OpenShift since 9.0.0 · Issue #8655 · elastic/cloud-on-k8s · GitHub but I'm a bit stuck for now.

Yes, the thing about $PATH is strange, because if I override the default container command with

tail -f /dev/null

and I enter the container, the $PATH is as you describe it and these components (APM, Maps) can be started manually OK (though Maps only start if I'm in /usr/src/app/server/app where I need to cd first). I'm, however, not the OpenShift administrator and can't access its various settings so I don't know what can be the cause.