Kibana lost all configuration

[peoh (Peo Håkansson)]

I upgraded Kibana to the lastest version about a month ago and it has been working fine. Today however, when I tried to access Kibana I got this error in my browser:

{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occured."}

Ok, that's strange. So I restarted the service "systemctl restart kibana".

Now, when I navigate to Kibana, all configurations is lost. It asks me to add index patterns to see my data. If I navigate to Stack Monitoring, everything looks fine. ES & Logstash are running and everything is green. But navigating Kibana, all is lost, index patterns, dashboards, everything. It even lost my spaces. It basicly look lite a brand new installation.

So, maybe a corrupt index and Kibana created a new one? Which index could I look at to find the old index-patterns? I did not have them backed up..

Any ideas where to go from here?

Thanks!

[Nathan_Reese]

What version are running? What version did you upgrade from?

Kibana persisted state is stored in an Elasticsearch index specified by configuration setting kibana.index. The default value is .kibana. Are you setting kibana.index configuration? Did your .kibana index get removed from Elasticsearch?

[peoh (Peo Håkansson)]

Running version is 7.16.2, and I upgraded from 7.15.something. Not sure which.

The "Default index"-setting looks to be empty.

Just saw that my lifetime policies are still there, that's nice.

If I list indecies containing "kibana" it finds these (all green and open), the number is doc-count:

.kibana-event-log-7.16.2-000001   3
.kibana-event-log-7.16.2-000002    21
.kibana_1   129
.kibana_2   134
.kibana_3   1221
.kibana_7.16.2_001   72
.kibana_task_manager_1   2
.kibana_task_manager_2   3
.kibana_task_manager_7.16.2_001   17
.monitoring-kibana-7-2022.01.21   17280
.monitoring-kibana-7-2022.01.22   17278
.monitoring-kibana-7-2022.01.23   17280
.monitoring-kibana-7-2022.01.24   17278
.monitoring-kibana-7-2022.01.25   17280
.monitoring-kibana-7-2022.01.26   8360
.monitoring-kibana-7-2022.01.27   1500

Edit: I believe I ran 7.15.12 before.
Edit 2: Doc-count was off by one row.

[peoh (Peo Håkansson)]

If I do this query, I do see the names of visualizations that are missing.

GET /.kibana_3/_search
{
    "query": {
        "match_all": {}
    }
}

Do you think it would be safe to change the kibana.index-setting to .kibana_3?

[peoh (Peo Håkansson)]

Changing defaultIndex to .kibana_3 and restarting only reverts the setting back to blank.

[peoh (Peo Håkansson)]

My fault, I changed the wrong setting.

Solution:

nano /etc/kibana/kibana.yml
Added this row:
kibana.index: ".kibana_3"
Restarted Kibana

And now it's all back to normal. For some reason Kibana decided to create a new index for its settings and moved my old settings to .kibana_3. Maybe a bug?