What did you do?
I tried upgrading from 7.12.0 to 7.13.2
What did you expect to see?
I expected the version to go to 7.13.2
What did you see instead? Under which circumstances?
There was no change running kubectl get kibana
kibana-prod green 1 7.12.0 58d
Environment
- ECK version:
helm list -n elastic-system
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
elastic-operator elastic-system 1 2021-04-26 11:18:02.286692269 +0100 BST deployed eck-operator-1.5.0 1.5.0
- Kubernetes information:
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.6-eks-49a6c0", GitCommit:"49a6c0bf091506e7bafcdb1b142351b69363355a", GitTreeState:"clean", BuildDate:"2020-12-23T22:10:21Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
- Resource definition:
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana-prod
namespace: elastic-system
spec:
version: 7.13.2
count: 1
elasticsearchRef:
name: "elasticsearch-prod"
http:
service:
spec:
type: NodePort
podTemplate:
spec:
containers:
- name: kibana
resources:
limits:
memory: 1Gi
cpu: 1
nodeSelector:
geeiq/node-type: ops
- Logs: (not sure if these are relevant)
Kibana logs
{"type":"response","@timestamp":"2021-07-02T12:43:47+00:00","tags":[],"pid":7,"method":"get","statusCode":200,"req":{"url":"/login","method":"get","headers":{"host":"10.0.1.75:5601","user-agent":"kube-probe/1.19+","accept-encoding":"gzip","connection":"close"},"remoteAddress":"10.0.1.54","userAgent":"kube-probe/1.19+"},"res":{"statusCode":200,"responseTime":25,"contentLength":127797},"message":"GET /login 200 25ms - 124.8KB"}
{"type":"log","@timestamp":"2021-07-02T12:43:47+00:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7,"message":"Error: Cluster client cannot be used after it has been closed.\n at LegacyClusterClient.assertIsNotClosed (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:196:13)\n at LegacyClusterClient.callAsInternalUser (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:116:12)\n at sendBulkPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/lib/send_bulk_payload.js:22:18)\n at BulkUploader._onPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:209:43)\n at BulkUploader._fetchAndUpload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:195:20)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:93:5)"}
{"type":"log","@timestamp":"2021-07-02T12:43:47+00:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7,"message":"Unable to bulk upload the stats payload to the local cluster"}
{"type":"response","@timestamp":"2021-07-02T12:43:54+00:00","tags":["access:console"],"pid":7,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_aliases&method=GET","method":"post","headers":{"x-forwarded-for":"81.147.78.99","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.geeiq.com","x-amzn-trace-id":"Root=1-60df0a0a-0016a5636b227f2d76021b96","content-length":"0","sec-ch-ua":"\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"","accept":"text/plain, */*; q=0.01","kbn-xsrf":"kibana","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","kbn-version":"7.12.0","origin":"https://kibana.geeiq.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.geeiq.com/app/dev_tools","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.0.1.215","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","referer":"https://kibana.geeiq.com/app/dev_tools"},"res":{"statusCode":200,"responseTime":127},"message":"POST /api/console/proxy?path=_aliases&method=GET 200 127ms"}
{"type":"response","@timestamp":"2021-07-02T12:43:54+00:00","tags":["access:console"],"pid":7,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_template&method=GET","method":"post","headers":{"x-forwarded-for":"81.147.78.
{"type":"response","@timestamp":"2021-07-02T12:43:47+00:00","tags":[],"pid":7,"method":"get","statusCode":200,"req":{"url":"/login","method":"get","headers":{"host":"10.0.1.75:5601","user-agent":"kube-probe/1.19+","accept-encoding":"gzip","connection":"close"},"remoteAddress":"10.0.1.54","userAgent":"kube-probe/1.19+"},"res":{"statusCode":200,"responseTime":25,"contentLength":127797},"message":"GET /login 200 25ms - 124.8KB"}
{"type":"log","@timestamp":"2021-07-02T12:43:47+00:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7,"message":"Error: Cluster client cannot be used after it has been closed.\n at LegacyClusterClient.assertIsNotClosed (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:196:13)\n at LegacyClusterClient.callAsInternalUser (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:116:12)\n at sendBulkPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/lib/send_bulk_payload.js:22:18)\n at BulkUploader._onPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:209:43)\n at BulkUploader._fetchAndUpload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:195:20)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:93:5)"}
{"type":"log","@timestamp":"2021-07-02T12:43:47+00:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7,"message":"Unable to bulk upload the stats payload to the local cluster"}
{"type":"response","@timestamp":"2021-07-02T12:43:54+00:00","tags":["access:console"],"pid":7,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_aliases&method=GET","method":"post","headers":{"x-forwarded-for":"81.147.78.99","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.geeiq.com","x-amzn-trace-id":"Root=1-60df0a0a-0016a5636b227f2d76021b96","content-length":"0","sec-ch-ua":"\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"","accept":"text/plain, */*; q=0.01","kbn-xsrf":"kibana","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","kbn-version":"7.12.0","origin":"https://kibana.geeiq.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.geeiq.com/app/dev_tools","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.0.1.215","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","referer":"https://kibana.geeiq.com/app/dev_tools"},"res":{"statusCode":200,"responseTime":127},"message":"POST /api/console/proxy?path=_aliases&method=GET 200 127ms"}
{"type":"response","@timestamp":"2021-07-02T12:43:54+00:00","tags":["access:console"],"pid":7,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_template&method=GET","method":"post","headers":{"x-forwarded-for":"81.147.78.99","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.geeiq.com","x-amzn-trace-id":"Root=1-60df0a0a-6623df9a6b0ff7f500e1c726","content-length":"0","sec-ch-ua":"\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"","accept":"text/plain, */*; q=0.01","kbn-xsrf":"kibana","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","kbn-version":"7.12.0","origin":"https://kibana.geeiq.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.geeiq.com/app/dev_tools","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.0.4.229","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","referer":"https://kibana.geeiq.com/app/dev_tools"},"res":{"statusCode":200,"responseTime":143},"message":"POST /api/console/proxy?path=_template&method=GET 200 143ms"}
{"type":"response","@timestamp":"2021-07-02T12:43:54+00:00","tags":["access:console"],"pid":7,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_mapping&method=GET","method":"post","headers":{"x-forwarded-for":"81.147.78.99","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.geeiq.com","x-amzn-trace-id":"Root=1-60df0a0a-26b5299868624bfd51101816","content-length":"0","sec-ch-ua":"\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"","accept":"text/plain, */*; q=0.01","kbn-xsrf":"kibana","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","kbn-version":"7.12.0","origin":"https://kibana.geeiq.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.geeiq.com/app/dev_tools","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.0.1.54","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","referer":"https://kibana.geeiq.com/app/dev_tools"},"res":{"statusCode":200,"responseTime":215},"message":"POST /api/console/proxy?path=_mapping&method=GET 200 215ms"}
{"type":"response","@timestamp":"2021-07-02T12:43:57+00:00","tags":[],"pid":7,"method":"get","statusCode":200,"req":{"url":"/login","method":"get","headers":{"host":"10.0.1.75:5601","user-agent":"kube-probe/1.19+","accept-encoding":"gzip","connection":"close"},"remoteAddress":"10.0.1.54","userAgent":"kube-probe/1.19+"},"res":{"statusCode":200,"responseTime":16,"contentLength":127797},"message":"GET /login 200 16ms - 124.8KB"}
{"type":"log","@timestamp":"2021-07-02T12:43:57+00:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7,"message":"Error: Cluster client cannot be used after it has been closed.\n at LegacyClusterClient.assertIsNotClosed (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:196:13)\n at LegacyClusterClient.callAsInternalUser (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:116:12)\n at sendBulkPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/lib/send_bulk_payload.js:22:18)\n at BulkUploader._onPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:209:43)\n at BulkUploader._fetchAndUpload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:195:20)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:93:5)"}
Additional Information
I tried reverting back to 7.12.0, In my resource if i revert the version back and apply again i get a message telling me version downgrades are not supported. Which is strange because when i run kubectl get kibana it clearly shows its on 7.12.0
The Kibana "kibana-prod" is invalid: spec.version: Forbidden: Version downgrades are not supported
...
spec:
version: 7.12.0
...