Red status: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]

Hi pals,

A few days ago we got the following error on our elastic node:

[FORBIDDEN/12/index read-only / allow delete (api)]

We found out that more disk space is needed, since only 5% of the space had remained for the elastic node. Next, we added more disk space and now the elastic node has occupied only 35% of the space. With this in mind, we are still facing the same error message! After googling, I disabled the searchguard plugin and ran the following command:

POST /_settings
{
  "index.blocks.read_only_allow_delete": null
}

But faced the following error and the elastic search status is RED!

{
    "error": {
        "root_cause": [
            {
                "type": "cluster_block_exception",
                "reason": "blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"
            }
        ],
        "type": "master_not_discovered_exception",
        "reason": "ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];]",
        "caused_by": {
            "type": "cluster_block_exception",
            "reason": "blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"
        }
    },
    "status": 503
}

Here is elasticsearch.yml:

bootstrap.memory_lock: false
cluster.name: elasticsearch
http.port: 9200
node.data: true
node.ingest: true
node.master: true
node.max_local_storage_nodes: 1
node.name: ELASTIC-SRV1
path.data: D:\Program\ELK\data
path.logs: D:\Program\ELK\logs
transport.tcp.port: 9300
xpack.license.self_generated.type: basic
xpack.security.enabled: false

searchguard.disabled: true
logger.org.elasticsearch.gateway: TRACE

network.host: 0.0.0.0
http.cors.allow-origin: "*"
http.cors.enabled: true

And the following are the status of the elastic node:

{
  "name" : "ELASTIC-SRV1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "6.5.4",
    "build_flavor" : "unknown",
    "build_type" : "unknown",
    "build_hash" : "d2ef93d",
    "build_date" : "2018-12-17T21:17:40.758843Z",
    "build_snapshot" : false,
    "lucene_version" : "7.5.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
# GET /_cat/master
Pbra1KbPRzGKWxoIxPg0Lw <IP Address> <IP Address> ELASTIC-SRV1
# GET /_cat/health
1569738758 06:32:38 elasticsearch red 1 1 223 223 0 4 11954 5 2.5s 1.8%

Could you please help me with the problem?!

Did you run that command very soon after starting Elasticsearch? If you try again does it report the same error?

You should expect that message if Elasticsearch is still starting up, but the response to GET _cat/health you shared suggests it should work now.

I guess it was not soon. It was around 1 minute after starting the Elasticsearch. I am still getting the same error:

[2019-09-29T10:17:23,284][ERROR][o.e.x.m.c.i.IndexRecoveryCollector] [ELASTIC-SRV1] collector [index_recovery] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:166) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.action.admin.indices.recovery.TransportRecoveryAction.checkGlobalBlock(TransportRecoveryAction.java:114) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.action.admin.indices.recovery.TransportRecoveryAction.checkGlobalBlock(TransportRecoveryAction.java:52) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.<init>(TransportBroadcastByNodeAction.java:255) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:233) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:78) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:395) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:384) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1242) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:60) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.xpack.monitoring.collector.indices.IndexRecoveryCollector.doCollect(IndexRecoveryCollector.java:76) ~[x-pack-monitoring-6.5.4.jar:6.5.4]
	at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:87) [x-pack-monitoring-6.5.4.jar:6.5.4]
	at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:278) [x-pack-monitoring-6.5.4.jar:6.5.4]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_202]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_202]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.4.jar:6.5.4]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]

You mean, the red status is not important for now?

The error you shared in your second message doesn't look like it came from an API call. What happens if you try the POST /_settings command again?

I mean that your cluster reports 4 initializing shards, which indicates it is no longer in the state not recovered state.

The second message was extracted from the log file. When calling the

POST /_settings
{
  "index.blocks.read_only_allow_delete": null
}

I get

{
    "acknowledged": false
}

What does that mean?

It means something went wrong processing that setting update, perhaps a timeout, but maybe it went through in the end. There will be more information in the logs.

I removed all the indices and restarted the elasticsearch and now it works

Seems a bit drastic. I'm pretty sure there was a solution that didn't involve deleting everything.

We have had a similar issue.
Deleting all indices and all data folder does not solved it.
It was some network NIC configuration that have changed - don't know why but it enabled jumbo frames and some network packets stop flowing - reset to it original configuration and everything start working smoothly as always.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.