org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]

Elastic Stack Elasticsearch
1

Hi, after a server reboot when I try to start elasticsearch I get the following error:
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]

this is followed by a lot of additional errors:

[2022-07-19T09:14:56,304][ERROR][o.e.x.m.c.m.JobStatsCollector] [am-elastic-acc-n1] collector [job_stats] failed to collect data
[2022-07-19T09:14:58,478][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [am-elastic-acc-n1] collector [cluster_stats] failed to collect data
[2022-07-19T09:15:56,862][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [am-elastic-acc-n1] failed to execute [search] input for watch [GYOADuWdQzGTxrQOIxxDLQ_logstash_version_mismatch], reason [all shards failed]
[2022-07-19T09:15:57,277][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [am-elastic-acc-n1] failed to execute [search] input for watch [GYOADuWdQzGTxrQOIxxDLQ_xpack_license_expiration], reason [all shards failed]
[2022-07-19T09:15:57,280][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [am-elastic-acc-n1] failed to execute [search] input for watch [GYOADuWdQzGTxrQOIxxDLQ_elasticsearch_version_mismatch], reason [all shards failed]
[2022-07-19T09:15:57,281][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [am-elastic-acc-n1] failed to execute [search] input for watch [GYOADuWdQzGTxrQOIxxDLQ_elasticsearch_cluster_status], reason [all shards failed]
[2022-07-19T09:15:57,285][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [am-elastic-acc-n1] failed to execute [search] input for watch [GYOADuWdQzGTxrQOIxxDLQ_elasticsearch_nodes], reason [all shards failed]
[2022-07-19T09:15:57,977][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [am-elastic-acc-n1] failed to execute [search] input for watch [GYOADuWdQzGTxrQOIxxDLQ_kibana_version_mismatch], reason [all shards failed]

In the end if I keep elasticsearch running all my data is deleted

My environment has been running fine for months, I am running version 7.16.2

2

Can you share more of your Elasticsearch log?

3

Below some more logging

[2022-07-19T09:14:45,780][INFO ][o.e.n.Node               ] [am-elastic-acc-n1] started
[2022-07-19T09:14:46,093][INFO ][o.e.c.s.ClusterSettings  ] [am-elastic-acc-n1] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2022-07-19T09:14:46,424][WARN ][o.e.x.s.a.ApiKeyAuthenticator] [am-elastic-acc-n1] Authentication using apikey failed - apikey authentication for id LqEtmYABfN3Wvu8XkwEC encountered a failure
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.checkGlobalBlock(TransportSingleShardAction.java:112) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.<init>(TransportSingleShardAction.java:146) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.<init>(TransportSingleShardAction.java:130) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:98) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:51) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:190) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:577) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:26) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:26) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:85) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:87) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:128) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.16.2.jar:7.16.2]
        at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) [x-pack-security-7.16.2.jar:7.16.2]

and here some more

[2022-07-19T09:14:46,764][INFO ][o.e.l.LicenseService     ] [am-elastic-acc-n1] license [2cec4340-38dc-482c-9875-efa058f0d40f] mode [platinum] - valid
[2022-07-19T09:14:46,764][INFO ][o.e.x.s.a.Realms         ] [am-elastic-acc-n1] license mode is [platinum], currently licensed security realms are [reserved/reserved,native/native1,active_directory/apg_ad]
[2022-07-19T09:14:46,765][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [am-elastic-acc-n1] Active license is now [PLATINUM]; Security is enabled
[2022-07-19T09:14:46,769][INFO ][o.e.g.GatewayService     ] [am-elastic-acc-n1] recovered [142] indices into cluster_state
[2022-07-19T09:14:47,206][INFO ][o.e.x.i.IndexLifecycleTransition] [am-elastic-acc-n1] moving index [.watcher-history-13-2022.07.09] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [watch-history-ilm-policy]
[2022-07-19T09:14:47,208][INFO ][o.e.x.i.IndexLifecycleTransition] [am-elastic-acc-n1] moving index [.watcher-history-13-2022.07.04] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [watch-history-ilm-policy]
[2022-07-19T09:14:47,209][INFO ][o.e.x.i.IndexLifecycleTransition] [am-elastic-acc-n1] moving index [.watcher-history-13-2022.07.06] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [watch-history-ilm-policy]
[2022-07-19T09:14:47,210][INFO ][o.e.x.i.IndexLifecycleTransition] [am-elastic-acc-n1] moving index [.watcher-history-13-2022.07.03] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [watch-history-ilm-policy]
[2022-07-19T09:14:47,213][INFO ][o.e.x.i.IndexLifecycleTransition] [am-elastic-acc-n1] moving index [.watcher-history-13-2022.07.07] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [watch-history-ilm-policy]
[2022-07-19T09:14:47,213][INFO ][o.e.x.i.IndexLifecycleTransition] [am-elastic-acc-n1] moving index [.watcher-history-13-2022.07.08] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [watch-history-ilm-policy]
[2022-07-19T09:14:47,214][INFO ][o.e.x.i.IndexLifecycleTransition] [am-elastic-acc-n1] moving index [.watcher-history-13-2022.07.05] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [watch-history-ilm-policy]
[2022-07-19T09:14:55,617][INFO ][o.e.x.w.WatcherService   ] [am-elastic-acc-n1] reloading watcher, reason [new local watcher shard allocation ids], cancelled [0] queued tasks
[2022-07-19T09:14:56,304][ERROR][o.e.x.m.c.m.JobStatsCollector] [am-elastic-acc-n1] collector [job_stats] failed to collect data
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:725) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:412) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:757) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:509) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:320) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:256) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:466) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:211) ~[elasticsearch-7.16.
4

It looks like you have a platinum license? If so definitely raise a request with the Support team.

It does look like Elasticsearch is trying to recover, what is the status of it?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.