Es service not starting up due to error

Hi Team,

My cluster was working fine but all of sudden i am getting below errors in elastic log. es service is not getting failed.

[T06:13:20,213][ERROR][o.e.x.s.a.e.ReservedRealm] [es_1] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
        at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:147) ~[x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:605) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:231) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:109) [x-pack-security-7.16.2.jar:7.16.2]
        at 
        at

/usr/share/Elasticsearch/bin/Elasticsearch-keystore list

bootstrap.password
keystore.seed
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password

I am not sure what has happened but there was no activity.

Trying to understand below but not able to understand,

Thanks,

Hi Team,

Can someone please reply.

Hi Team,

Can someone please reply.

Thanks,

See this line

org.elasticsearch.action.UnavailableShardsException: 
  at least one primary shard for the index [.security-7] is unavailable

This means, that you lost data, that contained the security/user data. Have you shut down a node that contained that shard of data without distributing it first?

Checking the logs will help in this case.

Hi @spinscale,

Thanks for reply. This is single node cluster. The node is up from +200 days.

What can be the issue in this case?

Thanks,

Have you taken a look at the logs?

Hi @spinscale ,

Thanks for reply.

I have uploaded logs after starting es service at below link.

I can see below messages getting appeared again and again.

at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.elasticsearch.action.NoShardAvailableActionException
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:544) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:491) [elasticsearch-7.16.2.jar:7.16.2]
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:725) [elasticsearch-7.16.2.jar:7.16.2]

Can you please have a look?

Thanks,

Your attached logs does not contain anything about the security index. Or the message above and thus it is impossible to figure out anything. The shard exceptions are about a different index. Can you provide logs from the moment, where there were issues with the above mentioned index?