I have deleted all files manually from /var/lib/elasticsearch/nodes/0/indices/. Now when i restart Elasticsearch server.
[2023-11-17T09:20:22,899][INFO ][o.e.x.s.a.RealmsAuthenticator] [ip] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-11-17T09:20:23,721][ERROR][o.e.x.s.a.e.ReservedRealm] [ip] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
I don't have any backups for the snapshot any way i can make this working?
It seems your nodes are unable to store the system index (.security index). Try to change your master node role as data_content by adding data_content role in node_roles of your nodes YML file.
Refer the below post for more detail-
After changing the role UUID of the node will change so in such case for existing cluster you need to kill all elastic node process then change the node role and restart your other node which results your cluster up and after that you'll not face this issue.
incase of any doubt share your YML settings or if it resolves let us know for the same.
We need to do some addition in our Elastic documentation so that it'll help to increase the deep understanding for developers and users about Elastic System index. I can see we don't have any significant explanation of System Index (security index) which is crucial Onwards Elastic Version 8 in cluster setup with dedicated master node.
There are lot of exceptions and errors available related to this concept directly or indirectly. Kindly ask the Developers & Documentation team to make the changes.
Incase if it's already available kindly share the link for reference.