UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable

[rahul_sirugudi]

Hi,
I have deleted all files manually from /var/lib/elasticsearch/nodes/0/indices/. Now when i restart Elasticsearch server.

[2023-11-17T09:20:22,899][INFO ][o.e.x.s.a.RealmsAuthenticator] [ip] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-11-17T09:20:23,721][ERROR][o.e.x.s.a.e.ReservedRealm] [ip] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable

I don't have any backups for the snapshot any way i can make this working?

[rahul_sirugudi]

I have followed few steps mentioned on this 2 links.

Accidentally deleted .security index for x-pack

nm, it seems to have come back

and here

https://discuss.elastic.co/t/elk-7-8-0-two-node-cluster-at-least-one-primary-shard-for-the-index-security-7-is-unavailable/261031/3

but this 2 steps didn't helped me.

[DavidTurner]

@rahul_sirugudi

I don't have any backups for the snapshot any way i can make this working?

Nope, sorry, it sounds like you've destroyed all your data. See these docs for more information:

WARNING: Don’t modify anything within the data directory or run processes that might interfere with its contents...

[NIK2501nc (NIKHIL KUMAR CHAUDHARY)]

Hi @rahul_sirugudi

It seems your nodes are unable to store the system index (.security index). Try to change your master node role as data_content by adding data_content role in node_roles of your nodes YML file.

Refer the below post for more detail-

Failed to retrieve password hash for reserved user [elastic] org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable

failed to retrieve password hash for reserved user [elastic] org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable You'll get this issue when in your cluster, the nodes are unable to contain the system index ( like ".index, .security, .kibana etc." ) for resolving this issue- Add the node role as "data_content" the YML setting will be- node roles: [data_content, other node roles whichever you want separated by comma] For D…

After changing the role UUID of the node will change so in such case for existing cluster you need to kill all elastic node process then change the node role and restart your other node which results your cluster up and after that you'll not face this issue.

incase of any doubt share your YML settings or if it resolves let us know for the same.

Good luck!

[NIK2501nc (NIKHIL KUMAR CHAUDHARY)]

Hi @DavidTurner ,

We need to do some addition in our Elastic documentation so that it'll help to increase the deep understanding for developers and users about Elastic System index. I can see we don't have any significant explanation of System Index (security index) which is crucial Onwards Elastic Version 8 in cluster setup with dedicated master node.

There are lot of exceptions and errors available related to this concept directly or indirectly. Kindly ask the Developers & Documentation team to make the changes.
Incase if it's already available kindly share the link for reference.

Thanks!

[DavidTurner]

Although the index reporting the problem here happens to be a system index, this fact is not in any way relevant. You deleted all the data in the data path.