Hi,
I am upgrading ELK stack from 6.6.1 to 7.8.0(oss) in kubernetes environment. Elasticsearch upgrade works fine, but new kibana pod fails with this error -
{"type":"log","@timestamp":"2020-08-16T18:01:43Z","tags":["warning","plugins-discovery"],"pid":1,"message":"Expect plugin "id" in camelCase, but found: apm_oss"}
{"type":"log","@timestamp":"2020-08-16T18:01:44Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin "visTypeXy" is disabled."}
{"type":"log","@timestamp":"2020-08-16T18:01:45Z","tags":["info","plugins-system"],"pid":1,"message":"Setting up [42] plugins: [usageCollection,telemetryCollectionManager,telemetry,kibanaLegacy,devTools,uiActions,statusPage,share,newsfeed,kibanaUtils,mapsLegacy,kibanaReact,inspector,embeddable,indexPatternManagement,esUiShared,discover,bfetch,expressions,data,home,apm_oss,console,management,advancedSettings,telemetryManagementSection,visualizations,visTypeVega,visTypeTimelion,visTypeTable,visTypeMarkdown,inputControlVis,savedObjects,navigation,visualize,dashboard,savedObjectsManagement,charts,visTypeVislib,visTypeTimeseries,visTypeMetric,visTypeTagcloud]"}
{"type":"log","@timestamp":"2020-08-16T18:01:45Z","tags":["info","savedobjects-service"],"pid":1,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2020-08-16T18:01:45Z","tags":["info","savedobjects-service"],"pid":1,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2020-08-16T18:01:45Z","tags":["info","savedobjects-service"],"pid":1,"message":"Creating index .kibana_1."}
{"type":"log","@timestamp":"2020-08-16T18:01:45Z","tags":["warning","savedobjects-service"],"pid":1,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_1/mQa9UmirST6QDtTeT6XjQQ] already exists, with { index_uuid="mQa9UmirST6QDtTeT6XjQQ" & index=".kibana_1" }"}
{"type":"log","@timestamp":"2020-08-16T18:01:45Z","tags":["warning","savedobjects-service"],"pid":1,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_1 and restarting Kibana."}
Before upgrade, there was a .kibana alias pointing to .kibana_1 index -
$ curl ip:9200/_cat/aliases?v
alias index filter routing.index routing.search
.kibana .kibana_1 - - -
$ curl ip:9200/_cat/indices
green open .kibana_1 mQa9UmirST6QDtTeT6XjQQ 1 0 2 0 12.7kb 12.7kb
After elasticsearch upgrade completes, I see strangely the alias is missing while the kibana_1 index remains.
$ curl ip:9200/_cat/aliases ## -> empty results
$ curl ip:9200/_cat/indices | grep kibana
green open .kibana_1 mQa9UmirST6QDtTeT6XjQQ 1 0 2 0 12.7kb 12.7kb
Just restarting kibana pod doesn't work, the only way I could resolve this was to delete .kibana_1 index & restart kibana. But with this, I lose all the saved objects stored in kibana.
Referred to - Kibana fails to start after upgrading from 7.2.1 to 7.5.0 · Issue #2352 · elastic/cloud-on-k8s · GitHub, it suggested to complete Elasticsearch upgrade before starting kibana upgrade. I tried that too, and ensured Elasticsearch got upgraded to 7.8.0 version & REST api is serviceable on 9200 port. After this, kibana upgrade was triggered.
But even then, the same exceptions are encountered (as shared above) and kibana upgrade fails.
Any suggestions to solve this and why this happens while upgrading kibana? And, why does the alias get deleted?
Also, if I run intermediate upgrades from ELK 6.6.1 -> 7.0.1 -> 7.8.0, kibana upgrade works fine. Any thoughts?