Hello all,
until recently I had Elastic stack running under ECK on OpenShift with no problems. However, after an upgrade to 9.0.0 (and now 9.0.1) APM server and Maps server stopped working, OpenShift emitting a weird error about executable not found in $PATH. So I did some manual digging in ECK-generated manifests and finally I went past the issues by modifying podTemplates for these resources like this:
APM server:
podTemplate:
spec:
containers:
- name: apm-server
env:
- name: PATH
value: "/bin:/usr/bin:/usr/share/apm-server"
volumeMounts:
- name: non-persistent
mountPath: /usr/share/apm-server/data
readOnly: false
volumes:
- name: non-persistent
emptyDir: {}
When finally starting the container, there was another error about not being able to create the data directory, hence the emptyDir.
Maps server:
podTemplate:
spec:
containers:
- name: maps
command:
- /bin/sh
- -c
- "cd /usr/src/app/server/app && node index.js"
So I've got it working. However, I hardly think this should be compulsory, or is it? I didn't find these issues documented anywhere. Is there a better way?
Thank you.
Hi,
I can't reproduce the issue for the APM server. Could you share the original manifest and your OpenShift version?
With regard to Maps I can confirm that the image does not start since 9.0.0
:
Warning Failed 17m (x9 over 18m) kubelet Error: container create failed: open executable: Operation not permitted
I'll create a GitHub issue in the ECK repo and try to understand the root cause.
Hello Michael and thank you for your response.
OpenShift version is 4.16.30 and I deployed APM server via Elastic helm chart with the following values.yaml before adding the podTemplate part:
fullnameOverride: "apm-server-poc"
version: 9.0.1
count: 1
config:
monitoring:
enabled: false
cluster_uuid: ZJn6TF45SOuEHCkdPCgVcQ
http:
enabled: true
port: 5066
host: 0.0.0.0
http:
service:
spec:
ports:
- name: http
port: 8200
targetPort: 8200
- name: metrics
port: 5066
targetPort: 5066
elasticsearchRef:
name: elastic-poc
kibanaRef:
name: kibana-poc
Regarding the Maps server I got a different error from yours. It was the same error as with the APM server about binary not being found in $PATH.
I also have APM and Maps server on another infrastructure where they run as containers as well (no k8s or OpenShift there, just Docker) and they work without problems.
Interesting, I'm using 4.18.1
.
For Maps the node
binary is in /usr/bin/node
, which is in the PATH
variable: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
This variable was not changed between 8.17
and 9.0.0
. So I don't understand why you would get an error about the binary not in $PATH
.
I created ElasticMapsServer does not start on OpenShift since 9.0.0 · Issue #8655 · elastic/cloud-on-k8s · GitHub but I'm a bit stuck for now.
Yes, the thing about $PATH
is strange, because if I override the default container command with
tail -f /dev/null
and I enter the container, the $PATH
is as you describe it and these components (APM, Maps) can be started manually OK (though Maps only start if I'm in /usr/src/app/server/app
where I need to cd
first). I'm, however, not the OpenShift administrator and can't access its various settings so I don't know what can be the cause.