Locked out after closing indexes to enable compression. :-(

Elastic Stack Elasticsearch
1

We have an Elasticsearch 7.17 4 node cluster and we were running low on space so I found an article on how to enable high compression on the indexes. The first step was to close the indexes. Big mistake because as soon as I ran the command I was locked out of Kibana and the ability to logon via SSH and change settings with curl. "unable to authenticate user [elastic] for REST request [/_cluster/health?pretty]" I can't do anything with curl as the elastic user is not able to be authenticated. I did a rolling restart of the cluster which did not help. Here's the article that I read and tried:
Apply best_compression to save disk space I'm hoping I dont' have to restore the 5 node cluster from backup. Please let me know if you have any ideas? Thanks in advance. JK

2

Yeah, this is complicated as the security index was closed as well and to open you would need to be able to authenticate, which is not possible.

One thing that you may try is to stop all nodes in your cluster, disable security, restart them, open the indices and them stop the cluster a second time to re-enable security.

Not sure if disabling security in just one node and making requests to this nodes would work, but you may try this as well.

1 Like
3

Thanks so much, I'll give it a try. JK

4

I would suggest you try the file realm:

The file realm is very useful as a fallback or recovery realm. For example in cases where [...] the security index is unavailable [...]

1 Like
5

Thanks again, That worked great. So glad that I don't have to restore 4 VMs with 1TB each. Sure appreciate the suggestion. :slight_smile:

6

Thanks for the reply David. Fortunately the previous suggestion worked for me. I will look over your link on the File realm. JK

7

:+1: glad you're up and running again. It's worth bearing the file realm in mind in future cos sometimes it's a bit of a pain to disable security everywhere. In an emergency you can even just set up the file realm on one node temporarily while you fix the problem. This realm is enabled by default so you don't have to restart anything, just update some config files, but even if you've disabled it explicitly, it's only one node to restart to bring it back again.