# ES2.0 globalBlockedException

**URL:** <https://discuss.elastic.co/t/es2-0-globalblockedexception/35189>\
**Category:** Elasticsearch\
**Created:** [November 20, 2015, 9:27pm UTC](https://discuss.elastic.co/t/es2-0-globalblockedexception/35189 "2015-11-20T21:27:37Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![mad\_min](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mad_min/32/5964_2.png) [@mad\_min](https://discuss.elastic.co/u/mad_min)\
**Post date:** [November 20, 2015, 9:27pm UTC](https://discuss.elastic.co/t/es2-0-globalblockedexception/35189/1 "2015-11-20T21:27:37Z")

</div>

Trying to send LS2.0 output to ES2.0 seeing cluster globalBlocked exceptions of these kinds:

```
    [2015-11-20 12:27:54,436][INFO][rest.suppressed] /_template/logstash Params: {name=logstash}
    ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];]
            at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:154)
            at org.elasticsearch.action.admin.indices.template.get.TransportGetIndexTemplatesAction.checkBlock(TransportGetIndexTemplatesAction.java:58)
            at org.elasticsearch.action.admin.indices.template.get.TransportGetIndexTemplatesAction.checkBlock(TransportGetIndexTemplatesAction.java:43)
            at org.elasticsearch.action.support.master.TransportMasterNodeAction.innerExecute(TransportMasterNodeAction.java:94)
            at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:86)
            at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:48)
            at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:70)
            at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)
            at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:347)
            at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52)
...  
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

```

And then

```
[2015-11-20 12:28:55,530][INFO][rest.suppressed] /_bulk Params: {}
ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:154)
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:144)
        at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:207)
        at org.elasticsearch.action.bulk.TransportBulkAction.access$000(TransportBulkAction.java:66)
        at org.elasticsearch.action.bulk.TransportBulkAction$1.onFailure(TransportBulkAction.java:145)
        at org.elasticsearch.action.support.ThreadedActionListener$2.doRun(ThreadedActionListener.java:104)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

```

My config on the ES2.0

```
cluster.name: live
node.name: srv-9
path.data: /home/s/elastic-data
path.logs: /home/s/elastic-logs
bootstrap.mlockall: true
gateway.recover_after_nodes: 8

discovery.zen.ping.unicast.hosts: ["10.86.205.55:9300", "10.86.205.60:9300"]
discovery.zen.minimum_master_nodes: 1
discovery.zen.fd.ping_timeout: 30s
discovery.zen.ping.multicast.enabled: false

action.disable_delete_all_indices: true

index.number_of_shards: 20
index.number_of_replicas: 3

network.bind_host: "0.0.0.0"
network.publish_host: _non_loopback:ipv4_

```

Logstash2 output config:

```
output {
 elasticsearch { 
 hosts => ["10.86.205.62:9200"] 
}
 stdout { codec => rubydebug }
}

```

Any thoughts or pointers?

I have tried restarting both ES and LS. It seems like LS is sending a big burst of messages. Thats because its reading from Redis which already has several messages on store when LS is coming up.

Thanks

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 21, 2015, 5:59am UTC](https://discuss.elastic.co/t/es2-0-globalblockedexception/35189/2 "2015-11-21T05:59:22Z")

</div>

What state is your ES cluster in?

---

<div class="post-metadata">

**Author:** ![mad\_min](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mad_min/32/5964_2.png) [@mad\_min](https://discuss.elastic.co/u/mad_min)\
**Post date:** [November 22, 2015, 10:16pm UTC](https://discuss.elastic.co/t/es2-0-globalblockedexception/35189/3 "2015-11-22T22:16:18Z")

</div>

Hi @warkolm, since my posting, I have removed the other hosts from the cluster and trying to run this standalone with one master/data. Still see the same issue. This is what I see:

```
{
  "cluster_name" : "live",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : "NaN"
}

```

and 'pending tasks' shows:

{  
"error" : {  
"root\_cause" : [ {  
"type" : "cluster\_block\_exception",  
"reason" : "blocked by: [SERVICE\_UNAVAILABLE/1/state not recovered / initialized];"  
} ],  
"type" : "cluster\_block\_exception",  
"reason" : "blocked by: [SERVICE\_UNAVAILABLE/1/state not recovered / initialized];"  
},  
"status" : 503  
}

Thanks!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 22, 2015, 11:35pm UTC](https://discuss.elastic.co/t/es2-0-globalblockedexception/35189/4 "2015-11-22T23:35:30Z")

</div>

It's pretty weird that it's red with no shards.  
Is there data on disk anywhere?

---

<div class="post-metadata">

**Author:** ![mad\_min](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mad_min/32/5964_2.png) [@mad\_min](https://discuss.elastic.co/u/mad_min)\
**Post date:** [November 22, 2015, 11:39pm UTC](https://discuss.elastic.co/t/es2-0-globalblockedexception/35189/5 "2015-11-22T23:39:03Z")

</div>

No data anywhere. This is a new setup where I am trying to bring up LS2 talking to ES2.

LS2 is getting data from Redis and as soon as LS2 comes up, it seems to send a huge burst of data to ES2, which ends up getting stuck with this exception. LS2 eventually complains its pipeline is full.

Thanks.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 22, 2015, 11:41pm UTC](https://discuss.elastic.co/t/es2-0-globalblockedexception/35189/6 "2015-11-22T23:41:28Z")

</div>

I'd stop everything and then bring ES up, make sure it is in a green state and then create some test indices and see what happens.

---

<div class="post-metadata">

**Author:** ![mad\_min](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mad_min/32/5964_2.png) [@mad\_min](https://discuss.elastic.co/u/mad_min)\
**Post date:** [November 23, 2015, 6:16am UTC](https://discuss.elastic.co/t/es2-0-globalblockedexception/35189/7 "2015-11-23T06:16:21Z")

</div>

Was getting the same issue after shutting everything down and bringing up ES2. Removing the following config got this to work correctly.

```
#discovery.zen.minimum_master_nodes: 1
#discovery.zen.fd.ping_timeout: 30s
#index.number_of_shards: 20
#index.number_of_replicas: 3

```

I guess it kind of makes sense that there should be no replicas or multiple shards in a one machine show. Dont know whether this is really the cause of whether there should be some error handling of the config. Just a lucky guess at this point from my side.

Thanks again!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:36pm UTC](https://discuss.elastic.co/t/es2-0-globalblockedexception/35189/8 "2017-07-05T23:36:40Z")

</div>

Closed on July 5, 2017, 11:36pm UTC