Kibana upgrade from 6.6.1 to 7.8.0 fails with resource already exists exception

Hi,

I am upgrading ELK stack from 6.6.1 to 7.8.0(oss) in kubernetes environment. Elasticsearch upgrade works fine, but new kibana pod fails with this error -

{"type":"log","@timestamp":"2020-08-16T18:01:43Z","tags":["warning","plugins-discovery"],"pid":1,"message":"Expect plugin "id" in camelCase, but found: apm_oss"}
{"type":"log","@timestamp":"2020-08-16T18:01:44Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin "visTypeXy" is disabled."}
{"type":"log","@timestamp":"2020-08-16T18:01:45Z","tags":["info","plugins-system"],"pid":1,"message":"Setting up [42] plugins: [usageCollection,telemetryCollectionManager,telemetry,kibanaLegacy,devTools,uiActions,statusPage,share,newsfeed,kibanaUtils,mapsLegacy,kibanaReact,inspector,embeddable,indexPatternManagement,esUiShared,discover,bfetch,expressions,data,home,apm_oss,console,management,advancedSettings,telemetryManagementSection,visualizations,visTypeVega,visTypeTimelion,visTypeTable,visTypeMarkdown,inputControlVis,savedObjects,navigation,visualize,dashboard,savedObjectsManagement,charts,visTypeVislib,visTypeTimeseries,visTypeMetric,visTypeTagcloud]"}
{"type":"log","@timestamp":"2020-08-16T18:01:45Z","tags":["info","savedobjects-service"],"pid":1,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2020-08-16T18:01:45Z","tags":["info","savedobjects-service"],"pid":1,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2020-08-16T18:01:45Z","tags":["info","savedobjects-service"],"pid":1,"message":"Creating index .kibana_1."}
{"type":"log","@timestamp":"2020-08-16T18:01:45Z","tags":["warning","savedobjects-service"],"pid":1,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_1/mQa9UmirST6QDtTeT6XjQQ] already exists, with { index_uuid="mQa9UmirST6QDtTeT6XjQQ" & index=".kibana_1" }"}
{"type":"log","@timestamp":"2020-08-16T18:01:45Z","tags":["warning","savedobjects-service"],"pid":1,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_1 and restarting Kibana."}

Before upgrade, there was a .kibana alias pointing to .kibana_1 index -

$ curl ip:9200/_cat/aliases?v
alias index filter routing.index routing.search
.kibana .kibana_1 - - -

$ curl ip:9200/_cat/indices
green open .kibana_1 mQa9UmirST6QDtTeT6XjQQ 1 0 2 0 12.7kb 12.7kb

After elasticsearch upgrade completes, I see strangely the alias is missing while the kibana_1 index remains.

$ curl ip:9200/_cat/aliases ## -> empty results

$ curl ip:9200/_cat/indices | grep kibana
green open .kibana_1 mQa9UmirST6QDtTeT6XjQQ 1 0 2 0 12.7kb 12.7kb

Just restarting kibana pod doesn't work, the only way I could resolve this was to delete .kibana_1 index & restart kibana. But with this, I lose all the saved objects stored in kibana.

Referred to - https://github.com/elastic/cloud-on-k8s/issues/2352, it suggested to complete Elasticsearch upgrade before starting kibana upgrade. I tried that too, and ensured Elasticsearch got upgraded to 7.8.0 version & REST api is serviceable on 9200 port. After this, kibana upgrade was triggered.
But even then, the same exceptions are encountered (as shared above) and kibana upgrade fails.

Any suggestions to solve this and why this happens while upgrading kibana? And, why does the alias get deleted?

Also, if I run intermediate upgrades from ELK 6.6.1 -> 7.0.1 -> 7.8.0, kibana upgrade works fine. Any thoughts?

Hey,

Our recommendation is to upgrade from 6.x to 7.0 then to 7.8.1 which should work fine. The upgrade assistant introduced in 6.8 should help you prepare for any further upgrades.

https://www.elastic.co/guide/en/kibana/current/upgrade.html#upgrade-6x

If you are doing an upgrade to 7.8 please try to upgrade to the latest patch 7.8.1 as recommended by the release notes: https://www.elastic.co/guide/en/kibana/7.8/release-notes-7.8.1.html

Hello @Bamieh,
I use elasticsearch & kibana - oss stack. The Upgrade Assistant in 6.8 is X-pack I believe and hence I cannot use it - https://www.elastic.co/guide/en/kibana/6.8/upgrade-assistant.html.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.