Can't delete or recover .kibana_security_session_1 index

[RRGTHWAR]

My storage team badly botched an upgrade, and as a result the .kibana_security_session_1 index in my ECK cluster was corrupted. I don't have any backups of it to restore, and I can't delete it because the superuser privilege in ECK doesn't allow deleting protected indices. So it's just sitting around unassigned because there are no good copies of it and making my cluster red. I can't log into Kibana because it keeps trying to use that index that doesn't exist. Is there any way to force Kibana to delete and recreate the index?

[RRGTHWAR]

Finally figured this out. In case anyone else lands here, below is what I did, formatted as curl commands since this problem was keeping me from getting into Kibana:

To delete protected indices in ECK, you need to create a role with the "delete_index" privilege and the "allow_restricted_indices" setting set to true, like so:

curl -X POST 'https://localhost:9200/_security/role/index_deletion' --key certificates/elasticsearch-ca.pem -k -u elastic -H 'Content-Type: application/json' -d '
{
    "cluster": ["all"],
    "indices": [
        {
            "names": ["*"],
            "privileges": ["delete_index"],
            "allow_restricted_indices": true
        }
    ]
}'

Then assign that to a user:

curl -X POST 'https://localhost:9200/_security/user/YOUR_USER' --key certificates/elasticsearch-ca.pem -k -u elastic -H 'Content-Type: application/json' -d '
{
    "roles": ["superuser", "index_deletion"],
    "password": "INSERT PASSWORD HERE"
}'

And finally, delete the index:

curl -X DELETE "https://localhost:9200/.kibana_security_session_1" --key certificates/elasticsearch-ca.pem -k -u YOUR_USER

Restart Kibana and it will recreate the index.

If anyone with some clout happens to see this, I understand why it's difficult to delete system indices, but it would be nice if there were a way to force Elasticsearch to recreate some of them from scratch. I didn't care about any of the sessions in this index, I just needed it to go away so that I could make a new one.

[William_Brafford]

Thanks for posting your findings. I've created a Github issue for this problem. Please feel free to add details there if I've missed anything. Add API for recreating the Kibana security session index · Issue #102107 · elastic/elasticsearch · GitHub