I have a system running Elasticsearch and Kibana 7.17.6. This is a single host with roughly 600 million items in 300 indices. The host OS is Ubuntu 22.04.
Earlier today a clumsy rsync command on my part stepped on some of the indices at a low level, including taking out some of the system indices, which blocked Kibana access. I tried moving forward to 8.4, found I have some very old 6.5.4 indices that need attention to do that, so I reinstalled 7.17.6. This system is a "hobby horse" for me, trying to return to the most recent backup would be ... agonizing, at best.
Elasticsearch came back, but no users. I used elasticsearch-user to make a superuser and I think Elasticsearch is doing what it should be, but there is only that one user and Kibana is eternally stuck in "Kibana server is not yet ready".
These are the system indices that are left after this misstep:
.monitoring-es-7-2022.10.12
.monitoring-kibana-7-2022.10.12
.apm-custom-link
.tasks
.fleet-policies-7
.geoip_databases
.security-7
.kibana_task_manager_7.17.6_001
.kibana_7.17.6_001
.apm-agent-configuration
There doesn't seem to be a way to change the password for the kibana_system account, so I changed /etc/kibana/kibana.yml to use the superuser account and password. This lets Kibana come up, lets me log in using that same name and password, but it then stops at the "Welcome to Elastic" login splash screen.
I encountered a problem like this with some users once before and the solution was just zapping them and making new ones. Now, with just one account on the system, I don't have a path to doing that. Is there a method to reset the Kibana portion of the install back to some sort of default so I can get in and rebuild the other things?