Es service not starting up due to error

[cool999 (cool)]

Hi Team,

My cluster was working fine but all of sudden i am getting below errors in elastic log. es service is not getting failed.

[T06:13:20,213][ERROR][o.e.x.s.a.e.ReservedRealm] [es_1] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
        at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:147) ~[x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:605) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:231) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:109) [x-pack-security-7.16.2.jar:7.16.2]
        at 
        at

/usr/share/elasticsearch/bin/elasticsearch-keystore list

bootstrap.password
keystore.seed
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password

I am not sure what has happened but there was no activity.

Trying to understand below but not able to understand,

Failed to retrieve password hash for reserved user [kibana]

What's the output of bin/elasticsearch-keystore list ? You would need to set the bootstrap.password to something with bin/elasticsearch-keystore -f add bootstrap.password Restart the node, as the keystore is read on startup and changes to its values are not refreshed on runtime. Run bin/elasticsearch-setup-passwords interactive You don't need the bootstrap.password anymore so you can remove it with bin/elasticsearch-keystore remove bootstrap.password

Accidentally deleted .security index for x-pack

For reference, in case anyone stumbles upon this thread, X-Pack automatically creates the .security index as needed, and you shouldn't need to reinstall the X-Pack plugin to resolve this sort of issue. Assuming you don't have a snapshot that you can/want to restore from, then the safest/easiest course of action is: Restart your cluster. This step will clear the security caches and make it easier to work with, and it will make sure the template for your security index is present and up to date…

Thanks,

[cool999 (cool)]

Hi Team,

Can someone please reply.

[cool999 (cool)]

Hi Team,

Can someone please reply.

Thanks,

[spinscale (Alexander Reelsen)]

See this line

org.elasticsearch.action.UnavailableShardsException: 
  at least one primary shard for the index [.security-7] is unavailable

This means, that you lost data, that contained the security/user data. Have you shut down a node that contained that shard of data without distributing it first?

Checking the logs will help in this case.

[cool999 (cool)]

Hi @spinscale,

Thanks for reply. This is single node cluster. The node is up from +200 days.

What can be the issue in this case?

Thanks,

[spinscale (Alexander Reelsen)]

Have you taken a look at the logs?

[cool999 (cool)]

Hi @spinscale ,

Thanks for reply.

I have uploaded logs after starting es service at below link.

es_log

I can see below messages getting appeared again and again.

at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.elasticsearch.action.NoShardAvailableActionException
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:544) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:491) [elasticsearch-7.16.2.jar:7.16.2]
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:725) [elasticsearch-7.16.2.jar:7.16.2]

Can you please have a look?

Thanks,

[spinscale (Alexander Reelsen)]

Your attached logs does not contain anything about the security index. Or the message above and thus it is impossible to figure out anything. The shard exceptions are about a different index. Can you provide logs from the moment, where there were issues with the above mentioned index?