Failed to retrieve password hash for reserved user [kibana] org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable

Elastic Stack Elasticsearch
1

We got issue with indices filling the spaces, we removed the indices and restarted the service but it doesn't seem to start properly. When checked elasticSearch log, I am getting the below exception

failed to retrieve password hash for reserved user [kibana]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable

Failed to get datafeed stats to include in ML usage
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

There are loads of exception in elasticsearch log file.

This is the health status and it is RED

CAT allocation
image

The .security-7 is in unassigned status along with others. When I check the reason and below is the outcome

"Cannot allocate status because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster"

Kibana and Elasticsearch are all deployed on single server and there is no multiple nodes.

There are no snapshots saved as well as it is missing repo path in elasticsearch.yml file.

Is there anyway to recover this and all of the unassigned shards please. Can someone help me with the issue.

2

This is the allocation status for .security-7 and I randomly checked for others and they seem to have failed for same reason

image
image1287×533 91.8 KB

3

How did you removed the indices? Did you removed them directly from the data path or you removed using requests to elasticsearch?

From what you shared it seems that your security index was manually removed.

4

We removed it manually from the server using the standard rm command in linux by selecting indices that are old

sudo rm -Rf /var/lib/elasticsearch/nodes/0/indices/8VQ4WXK_ReWqcadNuHamqg/

We tried restarting the server this morning and now we are getting this error when we try to access the elk url

"Kibana server is not ready yet"

Checking the kibana logs, this is what we get

request":{"method":"GET"},"response":{"body":{"bytes":619},"status_code":401,"headers":{"www-authenticate":"Basic realm="security" charset="UTF-8", Bearer realm="security", ApiKey","content-type":"application/json; charset=UTF-8","content-length":"619"}}},"url":{"path":"/_nodes"},"message":"401 - 619.0B\nGET /_nodes?filter_path=nodes..version%2Cnodes..http.publish_address%2Cnodes..ip [security_exception]: unable to authenticate user [kibana] for REST request [/_nodes?filter_path=nodes..version%2Cnodes..http.publish_address%2Cnodes..ip

5

This is your issue, you should never change the files used by elastisearch directly in the disk, everything needs to be done using the API.

Changing or delete data directly in the disk will break your cluster and make it unrecoverable, since you deleted those files, unfortunately your data is lost, you cannot recover it if you do not have a snapshot.

You will need to reconfigure your cluster from scratch, as it is a new empty cluster.

For this you need to stop everything, clean the data paths and start again.

1 Like
6

Thanks for your response. Can you give me some guidelines on how to clean the data paths please.

Steps I see

  1. Stop elasticsearch.service, apm-server.service, kibana.service
  2. clean datapaths?
  3. Start the above three services?

Would it cause any issue if I retry creating .security-7 as empty index using the reroute api? "allocate_empty_primary". Wanted to try that before giving up on getting it back