# Es service not starting up due to error

**URL:** <https://discuss.elastic.co/t/es-service-not-starting-up-due-to-error/302154>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-security\
**Created:** [April 11, 2022, 8:32pm UTC](https://discuss.elastic.co/t/es-service-not-starting-up-due-to-error/302154 "2022-04-11T20:32:03Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![cool999](https://avatars.discourse-cdn.com/v4/letter/c/d26b3c/32.png) [@cool999](https://discuss.elastic.co/u/cool999)\
**Post date:** [April 11, 2022, 8:32pm UTC](https://discuss.elastic.co/t/es-service-not-starting-up-due-to-error/302154/1 "2022-04-11T20:32:03Z")

</div>

Hi Team,

My cluster was working fine but all of sudden i am getting below errors in elastic log. es service is not getting failed.

```auto
[T06:13:20,213][ERROR][o.e.x.s.a.e.ReservedRealm] [es_1] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
        at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:147) ~[x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:605) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:231) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:109) [x-pack-security-7.16.2.jar:7.16.2]
        at 
        at

```

/usr/share/elasticsearch/bin/elasticsearch-keystore list

```auto
bootstrap.password
keystore.seed
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password

```

I am not sure what has happened but there was no activity.

Trying to understand below but not able to understand,

> [@Failed to retrieve password hash for reserved user \[kibana\]](https://discuss.elastic.co/t/failed-to-retrieve-password-hash-for-reserved-user-kibana/193592/5):
>
> What's the output of bin/elasticsearch-keystore list ? You would need to set the bootstrap.password to something with bin/elasticsearch-keystore -f add bootstrap.password Restart the node, as the keystore is read on startup and changes to its values are not refreshed on runtime. Run bin/elasticsearch-setup-passwords interactive You don't need the bootstrap.password anymore so you can remove it with bin/elasticsearch-keystore remove bootstrap.password

> [@Accidentally deleted .security index for x-pack](https://discuss.elastic.co/t/accidentally-deleted-security-index-for-x-pack/69844/3):
>
> For reference, in case anyone stumbles upon this thread, X-Pack automatically creates the .security index as needed, and you shouldn't need to reinstall the X-Pack plugin to resolve this sort of issue. Assuming you don't have a snapshot that you can/want to restore from, then the safest/easiest course of action is: Restart your cluster. This step will clear the security caches and make it easier to work with, and it will make sure the template for your security index is present and up to date…

Thanks,

---

<div class="post-metadata">

**Author:** ![cool999](https://avatars.discourse-cdn.com/v4/letter/c/d26b3c/32.png) [@cool999](https://discuss.elastic.co/u/cool999)\
**Post date:** [April 14, 2022, 6:20am UTC](https://discuss.elastic.co/t/es-service-not-starting-up-due-to-error/302154/2 "2022-04-14T06:20:26Z")

</div>

Hi Team,

Can someone please reply.

---

<div class="post-metadata">

**Author:** ![cool999](https://avatars.discourse-cdn.com/v4/letter/c/d26b3c/32.png) [@cool999](https://discuss.elastic.co/u/cool999)\
**Post date:** [April 19, 2022, 11:46am UTC](https://discuss.elastic.co/t/es-service-not-starting-up-due-to-error/302154/3 "2022-04-19T11:46:24Z")

</div>

Hi Team,

Can someone please reply.

Thanks,

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [April 20, 2022, 8:25am UTC](https://discuss.elastic.co/t/es-service-not-starting-up-due-to-error/302154/4 "2022-04-20T08:25:40Z")

</div>

See this line

```auto
org.elasticsearch.action.UnavailableShardsException: 
  at least one primary shard for the index [.security-7] is unavailable

```

This means, that you lost data, that contained the security/user data. Have you shut down a node that contained that shard of data without distributing it first?

Checking the logs will help in this case.

---

<div class="post-metadata">

**Author:** ![cool999](https://avatars.discourse-cdn.com/v4/letter/c/d26b3c/32.png) [@cool999](https://discuss.elastic.co/u/cool999)\
**Post date:** [April 20, 2022, 2:56pm UTC](https://discuss.elastic.co/t/es-service-not-starting-up-due-to-error/302154/5 "2022-04-20T14:56:20Z")

</div>

Hi @spinscale,

Thanks for reply. This is single node cluster. The node is up from +200 days.

What can be the issue in this case?

Thanks,

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [April 21, 2022, 9:12am UTC](https://discuss.elastic.co/t/es-service-not-starting-up-due-to-error/302154/6 "2022-04-21T09:12:42Z")

</div>

Have you taken a look at the logs?

---

<div class="post-metadata">

**Author:** ![cool999](https://avatars.discourse-cdn.com/v4/letter/c/d26b3c/32.png) [@cool999](https://discuss.elastic.co/u/cool999)\
**Post date:** [April 25, 2022, 8:29am UTC](https://discuss.elastic.co/t/es-service-not-starting-up-due-to-error/302154/7 "2022-04-25T08:29:19Z")

</div>

Hi @spinscale ,

Thanks for reply.

I have uploaded logs after starting es service at below link.

> **[es\_log](https://wetransfer.com/downloads/cb7a412af7ea0993034e420788a31aec20220425082705/86cc47)**
>
> 1 file sent via WeTransfer, the simplest way to send your files around the world

I can see below messages getting appeared again and again.

```auto
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.elasticsearch.action.NoShardAvailableActionException
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:544) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:491) [elasticsearch-7.16.2.jar:7.16.2]
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:725) [elasticsearch-7.16.2.jar:7.16.2]

```

Can you please have a look?

Thanks,

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [April 26, 2022, 8:34am UTC](https://discuss.elastic.co/t/es-service-not-starting-up-due-to-error/302154/8 "2022-04-26T08:34:15Z")

</div>

Your attached logs does not contain anything about the security index. Or the message above and thus it is impossible to figure out anything. The shard exceptions are about a different index. Can you provide logs from the moment, where there were issues with the above mentioned index?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 24, 2022, 8:34am UTC](https://discuss.elastic.co/t/es-service-not-starting-up-due-to-error/302154/9 "2022-05-24T08:34:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.