[SERVICE_UNAVAILABLE/1/state not recovered / initialized];

Elastic Stack Elasticsearch
1

Hello,

I have problem with my elasticsearch. I have only one node.
For several days elasticsearch does not want to collect data.

In logs I see this :

[2019-04-26T13:07:51,242][INFO ][o.e.n.Node               ] [BbThyoV] started
    [2019-04-26T13:07:54,834][WARN ][r.suppressed             ] path: /.kibana/doc/config%3A6.4.2, params: {index=.kibana, id=config:6.4.2, type=doc}
    org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
            at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:166) ~[elasticsearch-6.4.2.jar:6.4.2]
            at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.checkGlobalBlock(TransportSingleShardAction.java:105) ~[elasticsearch-6.4.2.jar:6.4.2]
            at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.<init>(TransportSingleShardAction.java:139) ~[elasticsearch-6.4.2.jar:6.4.2]
            at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.<init>(TransportSingleShardAction.java:123) ~[elasticsearch-6.4.2.jar:6.4.2]
            at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAc

or this :

[2019-04-26T13:22:34,136][INFO ][o.e.n.Node               ] [BbThyoV] started
[2019-04-26T13:22:35,333][WARN ][r.suppressed             ] path: /.reporting-*/esqueue/_search, params: {index=.reporting-*, type=esqueue, version=true}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:166) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:152) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:285) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.action.search.TransportSearchAction.lambda$doExecute$4(TransportSearchAction.java:192) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) [elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:114) [elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:87) [elasticsearch-6.4.2.jar:6.4.2]

Elasticsearch Version 6.4.2

My configuration :
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: XXX
http.port: 9200

http://xxx:9200/_cluster/health?pretty

{
  "cluster_name" : "elasticsearch",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 16,
  "active_shards" : 16,
  "relocating_shards" : 0,
  "initializing_shards" : 4,
  "unassigned_shards" : 427,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 4,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 3392,
  "active_shards_percent_as_number" : 3.5794183445190155
}

The whole infrastructure is: 1x elasticsearch, 1x logstash, 1x kibana, (I do not know how much) x filebeat

Please help

2

Can you share the whole Elasticsearch log from when the node last started? All this is telling us is that the cluster hasn't recovered, but it's not really saying why.

Also what does GET _cat/master return?

3

Logs from elasticsearch start : https://pastebin.com/zChqvEYj

http://elasticsearch-host:9200/_cat/master

BbThyoVKQ12tI11XoMUf6Q elasticsearch-host x.x.x.x BbThyoV

I am sorry but I had to change the host name to "elasticsearch-host" and the ip address to "x.x.x.x" because this is a company secret.

Thank you very much for your interest in my problem :slight_smile:

4

Ok, I can't work much more out from that log either. Could you add

logger.org.elasticsearch.gateway: TRACE

to your elasticsearch.yml, restart this node, and share the resulting logs?

5

There is log : https://pastebin.com/MJRm416i

6

This log ends a few seconds after the node started to recover, and it looks like recovery is working ok (if a little slower than I might expect):

[2019-04-29T14:06:19,088][TRACE][o.e.g.GatewayService     ] [BbThyoV] successful state recovery, importing cluster state...
[2019-04-29T14:06:22,568][TRACE][o.e.g.GatewayAllocator$InternalPrimaryShardAllocator] [BbThyoV] [.kibana][0] fetching [shard_started] from [{BbThyoV}{BbThyoVKQ12tI11XoMUf6Q}{zRurY4BmSy-kN3tjBSl-QQ}{elasticsearch-host}{x.x.x.x:9300}{ml.ma
[2019-04-29T14:06:22,625][TRACE][o.e.g.TransportNodesListGatewayStartedShards] [BbThyoV] [.kibana][0] loading local shard state info

Can you share more of this log?

7

I restarty my elasticsearch again and it start work :slight_smile: I see my data.

I collect log from this operation : https://oc.blue.pl/index.php/s/UYFysON2fBSbQpd/download

But I still don't understand what happened.

Now my logstash has some problem :

[2019-04-30T16:38:17,907][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,907][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,908][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,908][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,910][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,910][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,911][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,911][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,912][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,912][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,912][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,912][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,912][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,913][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,913][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,913][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,914][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,915][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,916][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,917][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,918][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,918][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,919][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,919][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,920][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,920][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,920][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-30T16:38:17,920][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>50}

I restarted logstash but its not work :frowning:

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.