Kibana not upgrading from 7.12.0 to 7.13.2

What did you do?

I tried upgrading from 7.12.0 to 7.13.2

What did you expect to see?

I expected the version to go to 7.13.2

What did you see instead? Under which circumstances?

There was no change running kubectl get kibana

kibana-prod   green    1       7.12.0    58d

Environment

  • ECK version:

helm list -n elastic-system

NAME            	NAMESPACE     	REVISION	UPDATED                                	STATUS  	CHART             	APP VERSION
elastic-operator	elastic-system	1       	2021-04-26 11:18:02.286692269 +0100 BST	deployed	eck-operator-1.5.0	1.5.0 
  • Kubernetes information:
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.6-eks-49a6c0", GitCommit:"49a6c0bf091506e7bafcdb1b142351b69363355a", GitTreeState:"clean", BuildDate:"2020-12-23T22:10:21Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
  • Resource definition:
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-prod
  namespace: elastic-system
spec:
  version: 7.13.2
  count: 1
  elasticsearchRef:
    name: "elasticsearch-prod"
  http:
    service:
      spec:
        type: NodePort
  podTemplate:
    spec:
      containers:
      - name: kibana
        resources:
          limits:
            memory: 1Gi
            cpu: 1
      nodeSelector:
        geeiq/node-type: ops
  • Logs: (not sure if these are relevant)

Kibana logs

{"type":"response","@timestamp":"2021-07-02T12:43:47+00:00","tags":[],"pid":7,"method":"get","statusCode":200,"req":{"url":"/login","method":"get","headers":{"host":"10.0.1.75:5601","user-agent":"kube-probe/1.19+","accept-encoding":"gzip","connection":"close"},"remoteAddress":"10.0.1.54","userAgent":"kube-probe/1.19+"},"res":{"statusCode":200,"responseTime":25,"contentLength":127797},"message":"GET /login 200 25ms - 124.8KB"}
{"type":"log","@timestamp":"2021-07-02T12:43:47+00:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7,"message":"Error: Cluster client cannot be used after it has been closed.\n    at LegacyClusterClient.assertIsNotClosed (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:196:13)\n    at LegacyClusterClient.callAsInternalUser (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:116:12)\n    at sendBulkPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/lib/send_bulk_payload.js:22:18)\n    at BulkUploader._onPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:209:43)\n    at BulkUploader._fetchAndUpload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:195:20)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)"}
{"type":"log","@timestamp":"2021-07-02T12:43:47+00:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7,"message":"Unable to bulk upload the stats payload to the local cluster"}
{"type":"response","@timestamp":"2021-07-02T12:43:54+00:00","tags":["access:console"],"pid":7,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_aliases&method=GET","method":"post","headers":{"x-forwarded-for":"81.147.78.99","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.geeiq.com","x-amzn-trace-id":"Root=1-60df0a0a-0016a5636b227f2d76021b96","content-length":"0","sec-ch-ua":"\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"","accept":"text/plain, */*; q=0.01","kbn-xsrf":"kibana","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","kbn-version":"7.12.0","origin":"https://kibana.geeiq.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.geeiq.com/app/dev_tools","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.0.1.215","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","referer":"https://kibana.geeiq.com/app/dev_tools"},"res":{"statusCode":200,"responseTime":127},"message":"POST /api/console/proxy?path=_aliases&method=GET 200 127ms"}
{"type":"response","@timestamp":"2021-07-02T12:43:54+00:00","tags":["access:console"],"pid":7,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_template&method=GET","method":"post","headers":{"x-forwarded-for":"81.147.78.
{"type":"response","@timestamp":"2021-07-02T12:43:47+00:00","tags":[],"pid":7,"method":"get","statusCode":200,"req":{"url":"/login","method":"get","headers":{"host":"10.0.1.75:5601","user-agent":"kube-probe/1.19+","accept-encoding":"gzip","connection":"close"},"remoteAddress":"10.0.1.54","userAgent":"kube-probe/1.19+"},"res":{"statusCode":200,"responseTime":25,"contentLength":127797},"message":"GET /login 200 25ms - 124.8KB"}
{"type":"log","@timestamp":"2021-07-02T12:43:47+00:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7,"message":"Error: Cluster client cannot be used after it has been closed.\n    at LegacyClusterClient.assertIsNotClosed (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:196:13)\n    at LegacyClusterClient.callAsInternalUser (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:116:12)\n    at sendBulkPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/lib/send_bulk_payload.js:22:18)\n    at BulkUploader._onPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:209:43)\n    at BulkUploader._fetchAndUpload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:195:20)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)"}
{"type":"log","@timestamp":"2021-07-02T12:43:47+00:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7,"message":"Unable to bulk upload the stats payload to the local cluster"}
{"type":"response","@timestamp":"2021-07-02T12:43:54+00:00","tags":["access:console"],"pid":7,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_aliases&method=GET","method":"post","headers":{"x-forwarded-for":"81.147.78.99","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.geeiq.com","x-amzn-trace-id":"Root=1-60df0a0a-0016a5636b227f2d76021b96","content-length":"0","sec-ch-ua":"\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"","accept":"text/plain, */*; q=0.01","kbn-xsrf":"kibana","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","kbn-version":"7.12.0","origin":"https://kibana.geeiq.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.geeiq.com/app/dev_tools","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.0.1.215","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","referer":"https://kibana.geeiq.com/app/dev_tools"},"res":{"statusCode":200,"responseTime":127},"message":"POST /api/console/proxy?path=_aliases&method=GET 200 127ms"}
{"type":"response","@timestamp":"2021-07-02T12:43:54+00:00","tags":["access:console"],"pid":7,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_template&method=GET","method":"post","headers":{"x-forwarded-for":"81.147.78.99","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.geeiq.com","x-amzn-trace-id":"Root=1-60df0a0a-6623df9a6b0ff7f500e1c726","content-length":"0","sec-ch-ua":"\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"","accept":"text/plain, */*; q=0.01","kbn-xsrf":"kibana","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","kbn-version":"7.12.0","origin":"https://kibana.geeiq.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.geeiq.com/app/dev_tools","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.0.4.229","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","referer":"https://kibana.geeiq.com/app/dev_tools"},"res":{"statusCode":200,"responseTime":143},"message":"POST /api/console/proxy?path=_template&method=GET 200 143ms"}
{"type":"response","@timestamp":"2021-07-02T12:43:54+00:00","tags":["access:console"],"pid":7,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_mapping&method=GET","method":"post","headers":{"x-forwarded-for":"81.147.78.99","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.geeiq.com","x-amzn-trace-id":"Root=1-60df0a0a-26b5299868624bfd51101816","content-length":"0","sec-ch-ua":"\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"","accept":"text/plain, */*; q=0.01","kbn-xsrf":"kibana","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","kbn-version":"7.12.0","origin":"https://kibana.geeiq.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.geeiq.com/app/dev_tools","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.0.1.54","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","referer":"https://kibana.geeiq.com/app/dev_tools"},"res":{"statusCode":200,"responseTime":215},"message":"POST /api/console/proxy?path=_mapping&method=GET 200 215ms"}
{"type":"response","@timestamp":"2021-07-02T12:43:57+00:00","tags":[],"pid":7,"method":"get","statusCode":200,"req":{"url":"/login","method":"get","headers":{"host":"10.0.1.75:5601","user-agent":"kube-probe/1.19+","accept-encoding":"gzip","connection":"close"},"remoteAddress":"10.0.1.54","userAgent":"kube-probe/1.19+"},"res":{"statusCode":200,"responseTime":16,"contentLength":127797},"message":"GET /login 200 16ms - 124.8KB"}
{"type":"log","@timestamp":"2021-07-02T12:43:57+00:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7,"message":"Error: Cluster client cannot be used after it has been closed.\n    at LegacyClusterClient.assertIsNotClosed (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:196:13)\n    at LegacyClusterClient.callAsInternalUser (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:116:12)\n    at sendBulkPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/lib/send_bulk_payload.js:22:18)\n    at BulkUploader._onPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:209:43)\n    at BulkUploader._fetchAndUpload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:195:20)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)"}

Additional Information

I tried reverting back to 7.12.0, In my resource if i revert the version back and apply again i get a message telling me version downgrades are not supported. Which is strange because when i run kubectl get kibana it clearly shows its on 7.12.0

The Kibana "kibana-prod" is invalid: spec.version: Forbidden: Version downgrades are not supported

...
spec:
  version: 7.12.0
  ...

Hey @Kay_Khan, thanks for your question.

Can you provide operator logs? Also, at what version is your Elasticsearch (elasticsearch-prod)? ECK will not upgrade Kibana across minor versions if Elasticsearch is not yet upgraded (to at least 7.13.0 in this case).

Yep i found that this morning, i should also upgrade my elasticsearch to 7.13 in order for kibana to upgrade