Kibana 7.11.2 fails to start post upgrade

[parthmaniar (Parth D. Maniar)]

I just upgraded my stack to 7.11.2 and kibana is not starting. My elasticsearch status is green. I have rebooted the host and the problem persists.

Mar 11 09:59:01 hostname systemd[1]: kibana.service: State 'stop-sigterm' timed out. Killing.
Mar 11 09:59:01 hostname systemd[1]: kibana.service: Killing process 2645 (node) with signal SIGKILL.
Mar 11 09:59:01 hostname systemd[1]: kibana.service: Killing process 2669 (node) with signal SIGKILL.
Mar 11 09:59:01 hostname systemd[1]: kibana.service: Killing process 2670 (node) with signal SIGKILL.
Mar 11 09:59:01 hostname systemd[1]: kibana.service: Killing process 2671 (node) with signal SIGKILL.
Mar 11 09:59:01 hostname systemd[1]: kibana.service: Killing process 2672 (node) with signal SIGKILL.
Mar 11 09:59:01 hostname systemd[1]: kibana.service: Killing process 2673 (node) with signal SIGKILL.
Mar 11 09:59:01 hostname systemd[1]: kibana.service: Killing process 2674 (n/a) with signal SIGKILL.
Mar 11 09:59:01 hostname systemd[1]: kibana.service: Killing process 2676 (n/a) with signal SIGKILL.
Mar 11 09:59:01 hostname systemd[1]: kibana.service: Killing process 2677 (n/a) with signal SIGKILL.
Mar 11 09:59:01 hostname systemd[1]: kibana.service: Killing process 2678 (n/a) with signal SIGKILL.
Mar 11 09:59:01 hostname systemd[1]: kibana.service: Main process exited, code=killed, status=9/KILL
Mar 11 09:59:01 hostname systemd[1]: kibana.service: Failed with result 'timeout'.

Following are the last set of logs from kibana.log

{"type":"log","@timestamp":"2021-03-11T10:08:12+00:00","tags":["info","savedobjects-service"],"pid":3271,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2021-03-11T10:08:12+00:00","tags":["info","savedobjects-service"],"pid":3271,"message":"Creating index .kibana_7."}
{"type":"log","@timestamp":"2021-03-11T10:08:12+00:00","tags":["warning","savedobjects-service"],"pid":3271,"message":"Unable to connect to Elasticsearch. Error: resource_already_exists_exception"}
{"type":"log","@timestamp":"2021-03-11T10:08:12+00:00","tags":["warning","savedobjects-service"],"pid":3271,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_7 and restarting Kibana."}

There is no other Kibana instance that is operational in the cluster. What is the best way to get the system online?

[rashmi (kulkarni)]

@parthmaniar

deleting index .kibana_7 and restarting Kibana."}

Have u tried deleting index .kibana_7 and restarting Kibana ? Try this and let us know if it works .

Rashmi

[parthmaniar (Parth D. Maniar)]

Yes that is how I got Kibana to work. Is there a way to diagnose what happened and how this can be avoided?

[cheiligers]

@parthmaniar The last set of logs leads up to the problem that caused Kibana to not start and the reason was that there was an issue with migrating saved objects (visualizations, saved searches etc).
You can try using verbose logging to get a lot more info from the logs.

[rudolf (Rudolf Meijering)]

See Upgrade migrations | Kibana Guide [7.11] | Elastic for more information about the process and ways to avoid errors.

The first time an upgrade fails Kibana will log a fatal error, this is the root cause. Your logs show subsequent Kibana instances noticing another instance has already started an attempt and then simply waiting for the migration to complete.

[parthmaniar (Parth D. Maniar)]

Thank you very much. I will play with this next time

[parthmaniar (Parth D. Maniar)]

Thank you very much. Noted. I did find different configuration between Kibana instances which may have caused the issue. I'll note my experience during the next update.