# \[SERVICE\_UNAVAILABLE/1/state not recovered / initialized\]; **URL:** \ **Category:** Elasticsearch\ **Created:** [April 26, 2019, 11:58am UTC](https://discuss.elastic.co/t/service-unavailable-1-state-not-recovered-initialized/178639 "2019-04-26T11:58:02Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![mkalinowski](https://avatars.discourse-cdn.com/v4/letter/m/aeb1de/32.png) [@mkalinowski](https://discuss.elastic.co/u/mkalinowski)\ **Post date:** [April 26, 2019, 11:58am UTC](https://discuss.elastic.co/t/service-unavailable-1-state-not-recovered-initialized/178639/1 "2019-04-26T11:58:03Z")
Hello, I have problem with my elasticsearch. I have only one node. For several days elasticsearch does not want to collect data. In logs I see this : ``` [2019-04-26T13:07:51,242][INFO][o.e.n.Node] [BbThyoV] started [2019-04-26T13:07:54,834][WARN][r.suppressed] path: /.kibana/doc/config%3A6.4.2, params: {index=.kibana, id=config:6.4.2, type=doc} org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]; at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:166) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.checkGlobalBlock(TransportSingleShardAction.java:105) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.(TransportSingleShardAction.java:139) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.(TransportSingleShardAction.java:123) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAc ``` or this : ``` [2019-04-26T13:22:34,136][INFO][o.e.n.Node] [BbThyoV] started [2019-04-26T13:22:35,333][WARN][r.suppressed] path: /.reporting-*/esqueue/_search, params: {index=.reporting-*, type=esqueue, version=true} org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]; at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:166) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:152) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:285) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.action.search.TransportSearchAction.lambda$doExecute$4(TransportSearchAction.java:192) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) [elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:114) [elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:87) [elasticsearch-6.4.2.jar:6.4.2] ``` Elasticsearch Version 6.4.2 My configuration : path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: XXX http.port: 9200 [http://xxx:9200/\_cluster/health?pretty](http://xxx:9200/_cluster/health?pretty) ``` { "cluster_name" : "elasticsearch", "status" : "red", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 16, "active_shards" : 16, "relocating_shards" : 0, "initializing_shards" : 4, "unassigned_shards" : 427, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 4, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 3392, "active_shards_percent_as_number" : 3.5794183445190155 } ``` The whole infrastructure is: 1x elasticsearch, 1x logstash, 1x kibana, (I do not know how much) x filebeat Please help ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 26, 2019, 12:28pm UTC](https://discuss.elastic.co/t/service-unavailable-1-state-not-recovered-initialized/178639/2 "2019-04-26T12:28:47Z")
Can you share the whole Elasticsearch log from when the node last started? All this is telling us is that the cluster hasn't recovered, but it's not really saying why. Also what does `GET _cat/master` return? ---
**Author:** ![mkalinowski](https://avatars.discourse-cdn.com/v4/letter/m/aeb1de/32.png) [@mkalinowski](https://discuss.elastic.co/u/mkalinowski)\ **Post date:** [April 26, 2019, 3:43pm UTC](https://discuss.elastic.co/t/service-unavailable-1-state-not-recovered-initialized/178639/3 "2019-04-26T15:43:28Z")
Logs from elasticsearch start : [https://pastebin.com/zChqvEYj](https://pastebin.com/zChqvEYj) [http://elasticsearch-host:9200/\_cat/master](http://elasticsearch-host:9200/_cat/master) `BbThyoVKQ12tI11XoMUf6Q elasticsearch-host x.x.x.x BbThyoV` I am sorry but I had to change the host name to "elasticsearch-host" and the ip address to "x.x.x.x" because this is a company secret. Thank you very much for your interest in my problem 🙂 ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 26, 2019, 9:26pm UTC](https://discuss.elastic.co/t/service-unavailable-1-state-not-recovered-initialized/178639/4 "2019-04-26T21:26:51Z")
Ok, I can't work much more out from that log either. Could you add ```auto logger.org.elasticsearch.gateway: TRACE ``` to your `elasticsearch.yml`, restart this node, and share the resulting logs? ---
**Author:** ![mkalinowski](https://avatars.discourse-cdn.com/v4/letter/m/aeb1de/32.png) [@mkalinowski](https://discuss.elastic.co/u/mkalinowski)\ **Post date:** [April 29, 2019, 12:13pm UTC](https://discuss.elastic.co/t/service-unavailable-1-state-not-recovered-initialized/178639/5 "2019-04-29T12:13:07Z")
There is log : [https://pastebin.com/MJRm416i](https://pastebin.com/MJRm416i) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 30, 2019, 3:23am UTC](https://discuss.elastic.co/t/service-unavailable-1-state-not-recovered-initialized/178639/6 "2019-04-30T03:23:14Z")
This log ends a few seconds after the node started to recover, and it looks like recovery is working ok (if a little slower than I might expect): ```auto [2019-04-29T14:06:19,088][TRACE][o.e.g.GatewayService] [BbThyoV] successful state recovery, importing cluster state... [2019-04-29T14:06:22,568][TRACE][o.e.g.GatewayAllocator$InternalPrimaryShardAllocator] [BbThyoV] [.kibana][0] fetching [shard_started] from [{BbThyoV}{BbThyoVKQ12tI11XoMUf6Q}{zRurY4BmSy-kN3tjBSl-QQ}{elasticsearch-host}{x.x.x.x:9300}{ml.ma [2019-04-29T14:06:22,625][TRACE][o.e.g.TransportNodesListGatewayStartedShards] [BbThyoV] [.kibana][0] loading local shard state info ``` Can you share more of this log? ---
**Author:** ![mkalinowski](https://avatars.discourse-cdn.com/v4/letter/m/aeb1de/32.png) [@mkalinowski](https://discuss.elastic.co/u/mkalinowski)\ **Post date:** [April 30, 2019, 2:50pm UTC](https://discuss.elastic.co/t/service-unavailable-1-state-not-recovered-initialized/178639/7 "2019-04-30T14:50:47Z")
I restarty my elasticsearch again and it start work 🙂 I see my data. I collect log from this operation : [https://oc.blue.pl/index.php/s/UYFysON2fBSbQpd/download](https://oc.blue.pl/index.php/s/UYFysON2fBSbQpd/download) But I still don't understand what happened. * * * Now my logstash has some problem : ``` [2019-04-30T16:38:17,907][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,907][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,908][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,908][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,910][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,910][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,911][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,911][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,912][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,912][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,912][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,912][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,912][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,913][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,913][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,913][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,914][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,915][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,916][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,917][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,918][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,918][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,919][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,919][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,920][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,920][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,920][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2019-04-30T16:38:17,920][INFO][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>50} ``` I restarted logstash but its not work ☹ ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 28, 2019, 2:50pm UTC](https://discuss.elastic.co/t/service-unavailable-1-state-not-recovered-initialized/178639/8 "2019-05-28T14:50:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.