Failed to retrieve password hash for reserved user [kibana] org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable

Elastic Stack Elasticsearch
1

We got issue with indices filling the spaces, we removed the indices and restarted the service but it doesn't seem to start properly. When checked elasticSearch log, I am getting the below exception

failed to retrieve password hash for reserved user [kibana]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable

Failed to get datafeed stats to include in ML usage
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

There are loads of exception in elasticsearch log file.

This is the health status and it is RED

CAT allocation
image

The .security-7 is in unassigned status along with others. When I check the reason and below is the outcome

"Cannot allocate status because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster"

Kibana and Elasticsearch are all deployed on single server and there is no multiple nodes.

There are no snapshots saved as well as it is missing repo path in elasticsearch.yml file.

Is there anyway to recover this and all of the unassigned shards please. Can someone help me with the issue.

2

This is the allocation status for .security-7 and I randomly checked for others and they seem to have failed for same reason

image
image1287×533 91.8 KB

3

How did you removed the indices? Did you removed them directly from the data path or you removed using requests to elasticsearch?

From what you shared it seems that your security index was manually removed.