# Kibana crash on re-index of own index
**URL:** https://discuss.elastic.co/t/kibana-crash-on-re-index-of-own-index/236863
**Category:** Kibana
**Created:** [June 12, 2020, 9:49am UTC](https://discuss.elastic.co/t/kibana-crash-on-re-index-of-own-index/236863 "2020-06-12T09:49:58Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@raulgs](https://discuss.elastic.co/u/raulgs)
#### Post date: [June 12, 2020, 9:49am UTC](https://discuss.elastic.co/t/kibana-crash-on-re-index-of-own-index/236863/1 "2020-06-12T09:49:58Z")
Hi this morning I found my kibana pods in non-ready state with the following log entry
```auto
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["info","plugins","monitoring","monitoring"],"pid":7,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["warning","plugins","monitoring","monitoring"],"pid":7,"message":"X-Pack Monitoring Cluster Alerts will not be available: undefined"}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["info","savedobjects-service"],"pid":7,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["info","plugins","watcher"],"pid":7,"message":"Your basic license does not support watcher. Please upgrade your license."}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["info","savedobjects-service"],"pid":7,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["info","savedobjects-service"],"pid":7,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["info","savedobjects-service"],"pid":7,"message":"Reindexing .kibana to .kibana_1"}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["warning","savedobjects-service"],"pid":7,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_1/pzVPRvhORpiIt3wpMwJlCg] already exists, with { index_uuid=\"pzVPRvhORpiIt3wpMwJlCg\" & index=\".kibana_1\" }"}
{"type":"log","@timestamp":"2020-06-12T08:32:29Z","tags":["warning","savedobjects-service"],"pid":7,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_1 and restarting Kibana."}
```
After that I deleted .kibana\_2 which led to an error saying I should delete .kibana\_1.
Result was kibana did come up again however all kibana settings were lost. Safed searches, kibana advanced settings etc. Luckily this happend on our DEV cluster.
Does anybody know how it came to this issue? Have I misconfigured something??
My elastic stack is based on ECK am I am on version 7.7.1
Thanks in advance
---
### Author:  [@raulgs](https://discuss.elastic.co/u/raulgs)
#### Post date: [June 16, 2020, 10:18am UTC](https://discuss.elastic.co/t/kibana-crash-on-re-index-of-own-index/236863/2 "2020-06-16T10:18:45Z")
Nobody having an idea? 🤔
---
### Author:  [@rudolf](https://discuss.elastic.co/u/rudolf)
#### Post date: [June 16, 2020, 4:40pm UTC](https://discuss.elastic.co/t/kibana-crash-on-re-index-of-own-index/236863/3 "2020-06-16T16:40:13Z")
When you upgrade Kibana it might require a migration to migrate saved objects into the latest format. In your case it looks like there was a `.kibana` index, not an alias so it seems like the migration was from a 6.x version? What version did you upgrade from?
When several Kibana's are started up at the same time, only one of them will perform the migration. All the other Kibana pods will display this error "Waiting for that migration to complete.".
Are you sure another node wasn't still busy migrating the index? One of you nodes should have a log line like "Migrating .kibana\_1 saved objects to .kibana\_2" is there an error on this node? If not it might have just not had enough time to complete the migration.
---
### Author:  [@raulgs](https://discuss.elastic.co/u/raulgs)
#### Post date: [July 14, 2020, 10:46am UTC](https://discuss.elastic.co/t/kibana-crash-on-re-index-of-own-index/236863/4 "2020-07-14T10:46:15Z")
It seems like this issue happend because the kibana index has been set to read only by the index lifecycle policy. I will have to check on the index pattern. Wasn't aware that they also match on system policies...
---
### Author:  [@tiagocosta](https://discuss.elastic.co/u/tiagocosta)
#### Post date: [July 20, 2020, 8:36pm UTC](https://discuss.elastic.co/t/kibana-crash-on-re-index-of-own-index/236863/5 "2020-07-20T20:36:55Z")
@raulgs have you solved that problem? Usually it is not recommended to delete/edit any `.kibana_N` unless if you found problems in a migration process that usually happens in an upgrade situation as explained by @rudolf (at least do not change them without backing up your cluster).
Cheers
---
### Author:  [@raulgs](https://discuss.elastic.co/u/raulgs)
#### Post date: [July 21, 2020, 7:27am UTC](https://discuss.elastic.co/t/kibana-crash-on-re-index-of-own-index/236863/6 "2020-07-21T07:27:07Z")
@tiagocosta yes and I did. There was no other way I could get it up and running again. Thats I run a index delete command on the kibana\_1 index. After that I restarted kibana which did come up again. However, I unfortunately lost some saved searches. Not sure why ILM are being applied to system indicies as well. Makes no sense from my point of view...
Nowadays I have a snapshot being created every night.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 18, 2020, 7:37am UTC](https://discuss.elastic.co/t/kibana-crash-on-re-index-of-own-index/236863/7 "2020-08-18T07:37:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.