UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable

Elastic Stack Elasticsearch
1

Hi,
I have deleted all files manually from /var/lib/elasticsearch/nodes/0/indices/. Now when i restart Elasticsearch server.

[2023-11-17T09:20:22,899][INFO ][o.e.x.s.a.RealmsAuthenticator] [ip] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-11-17T09:20:23,721][ERROR][o.e.x.s.a.e.ReservedRealm] [ip] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable

I don't have any backups for the snapshot any way i can make this working?

2

I have followed few steps mentioned on this 2 links.

and here

https://discuss.elastic.co/t/elk-7-8-0-two-node-cluster-at-least-one-primary-shard-for-the-index-security-7-is-unavailable/261031/3

but this 2 steps didn't helped me.

3

Nope, sorry, it sounds like you've destroyed all your data. See these docs for more information:

WARNING: Don’t modify anything within the data directory or run processes that might interfere with its contents...

4

Hi @rahul_sirugudi

It seems your nodes are unable to store the system index (.security index). Try to change your master node role as data_content by adding data_content role in node_roles of your nodes YML file.

Refer the below post for more detail-

After changing the role UUID of the node will change so in such case for existing cluster you need to kill all elastic node process then change the node role and restart your other node which results your cluster up and after that you'll not face this issue.

incase of any doubt share your YML settings or if it resolves let us know for the same.

Good luck!

5

Hi @DavidTurner ,

We need to do some addition in our Elastic documentation so that it'll help to increase the deep understanding for developers and users about Elastic System index. I can see we don't have any significant explanation of System Index (security index) which is crucial Onwards Elastic Version 8 in cluster setup with dedicated master node.

There are lot of exceptions and errors available related to this concept directly or indirectly. Kindly ask the Developers & Documentation team to make the changes.
Incase if it's already available kindly share the link for reference.

Thanks!

6

Although the index reporting the problem here happens to be a system index, this fact is not in any way relevant. You deleted all the data in the data path.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.