KIbana "Kibana server is not ready yet" -- routing disabled during upgrade

Elastic Stack Kibana
1

Hi.
So I think I made the mistake of upgrading to v7.10 Kibana with my cluster routing disabled, and as a result Kibana is stuck migrating. It's constantly producing log messages along the line of;

["info", "savedobjects-service"], "pid":5709,"message":"Detected mapping change in \"properties.originId\""

Stopping Kibana and deleting the latest instance of .kibana_n and restarting the Kibana just produces another. I've also enabled routing etc.

I don't want to lose all the dashboards, but I'm not sure what the solution is, can anyone help. I've read lots of prior posts on this but there doesn't seem to be a clear solution.
thanks

1 Like
2

I also get this error after upgrading to 7.10.1

4

I had the same issue after ugprading to 7.10.1. The solution I came up with was this:

Before upgrading, export all saved objects at Stack Management > Saved objects.

Stop Kibana.

Delete the .kibana_* and .kibana_task_manager_* indexes:

$ for index in .kibana .kibana_1 .kibana_2 .kibana_3 .kibana_4 .kibana_task_manager_1 .kibana_task_manager_2 .kibana_task_manager_3; do curl -XDELETE http://localhost:9200/$index; done
{"acknowledged":true}{"acknowledged":true}...

Upgrade and start Kibana.

Reimport the previously exported saved objects at Stack Management > Saved objects. This worked pretty well. Despite some few errors about canvas-workpad-template stuff ("Unsupported object type"), all my dashboards, index patterns and so on will be there and working again. Not sure if the .kibana_task_manager_* really needed to be deleted, but it worked for me.

5

Every time Kibana is restarted, it checks if a migration is required. It will perform a migration if either documents are outdated or if the mappings are different to what Kibana expects. The latter is what triggers a migration in your case.

As per the error message, properties.originId has a different mapping to what Kibana expects. Do you have any index templates that might have an indexPattern that matches .kibana*?

Could you share the output of GET /.kibana/_mapping?

6

Hey, so I'm running into the same issue as mentioned by the original reporter. Luckily I have s3 snapshotting enabled on .kibana* so I have the liberty to experiment back and forth. Doing the export/import procedure does not cut it for me. I end up with missing dashboards unfortunately. If pasted the mapping on pastebin: https://pastebin.com/yrGSbmg6

I hope this provides some clue as to what's going on here. I'm upgrading from 7.8.0.

7

@IslandMan could you restore your snapshot, and then attempt a migration and post all of the logs from the first time the upgrade migration runs? Please open a new thread and ping me.

1 Like
8

If anyone else stumbles on this thread, @IslandMan's issue was solved in Upgrading Kibana to 7.10.2 hangs while migrating

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.