Kibana crash on re-index of own index

[raulgs (Raúl)]

Hi this morning I found my kibana pods in non-ready state with the following log entry

{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["info","plugins","monitoring","monitoring"],"pid":7,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["warning","plugins","monitoring","monitoring"],"pid":7,"message":"X-Pack Monitoring Cluster Alerts will not be available: undefined"}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["info","savedobjects-service"],"pid":7,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["info","plugins","watcher"],"pid":7,"message":"Your basic license does not support watcher. Please upgrade your license."}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["info","savedobjects-service"],"pid":7,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["info","savedobjects-service"],"pid":7,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["info","savedobjects-service"],"pid":7,"message":"Reindexing .kibana to .kibana_1"}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["warning","savedobjects-service"],"pid":7,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_1/pzVPRvhORpiIt3wpMwJlCg] already exists, with { index_uuid=\"pzVPRvhORpiIt3wpMwJlCg\" & index=\".kibana_1\" }"}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["warning","savedobjects-service"],"pid":7,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_1 and restarting Kibana."}

After that I deleted .kibana_2 which led to an error saying I should delete .kibana_1.
Result was kibana did come up again however all kibana settings were lost. Safed searches, kibana advanced settings etc. Luckily this happend on our DEV cluster.

Does anybody know how it came to this issue? Have I misconfigured something??
My elastic stack is based on ECK am I am on version 7.7.1

Thanks in advance

[raulgs (Raúl)]

Nobody having an idea?

[rudolf (Rudolf Meijering)]

When you upgrade Kibana it might require a migration to migrate saved objects into the latest format. In your case it looks like there was a .kibana index, not an alias so it seems like the migration was from a 6.x version? What version did you upgrade from?

When several Kibana's are started up at the same time, only one of them will perform the migration. All the other Kibana pods will display this error "Waiting for that migration to complete.".

Are you sure another node wasn't still busy migrating the index? One of you nodes should have a log line like "Migrating .kibana_1 saved objects to .kibana_2" is there an error on this node? If not it might have just not had enough time to complete the migration.

[raulgs (Raúl)]

It seems like this issue happend because the kibana index has been set to read only by the index lifecycle policy. I will have to check on the index pattern. Wasn't aware that they also match on system policies...

[tiagocosta]

@raulgs have you solved that problem? Usually it is not recommended to delete/edit any .kibana_N unless if you found problems in a migration process that usually happens in an upgrade situation as explained by @rudolf (at least do not change them without backing up your cluster).

Cheers

[raulgs (Raúl)]

@tiagocosta yes and I did. There was no other way I could get it up and running again. Thats I run a index delete command on the kibana_1 index. After that I restarted kibana which did come up again. However, I unfortunately lost some saved searches. Not sure why ILM are being applied to system indicies as well. Makes no sense from my point of view...

Nowadays I have a snapshot being created every night.