# Discovery.zen.publish reports pending node, cluster gets red at index create

**URL:** <https://discuss.elastic.co/t/discovery-zen-publish-reports-pending-node-cluster-gets-red-at-index-create/24612>\
**Category:** Elasticsearch\
**Created:** [June 30, 2015, 9:19am UTC](https://discuss.elastic.co/t/discovery-zen-publish-reports-pending-node-cluster-gets-red-at-index-create/24612 "2015-06-30T09:19:04Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Suny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suny/32/22082_2.png) [@Suny](https://discuss.elastic.co/u/Suny)\
**Post date:** [June 30, 2015, 9:19am UTC](https://discuss.elastic.co/t/discovery-zen-publish-reports-pending-node-cluster-gets-red-at-index-create/24612/1 "2015-06-30T09:19:04Z")

</div>

Hi. Our cluster went red a week ago. Sorry in advance, I wasn't able to investigate any better.  
In the evening, the cluster got a pending node:  
[2015-06-23 21:12:39,839][WARN][discovery.zen.publish] [xx104] timed out waiting for all nodes to process published state [649] (timeout [30s], pending nodes: [[xx114-y2][9YHYIAl6T82ZuMfg5a7oaA][xx114-y2][inet[/x.x.x.x:9300]]{disk\_type=ssd, machine\_id=es114, master=false}])  
This message was repeated every couple of minutes until [2015-06-24 04:26:03,513], when the node was restarted.  
Some hours later, logstash tried to create a new index. At this point, the cluster went red.  
[2015-06-24 02:01:01,193][DEBUG][action.admin.indices.create] [xx104] [logstash-2015.06.24] failed to create

org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (acquire index lock) within 1m

```
    at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.doRun(MetaDataCreateIndexService.java:150)

    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:745)

```

Then the pending node was restarted and the cluster went green again.  
I couldn't find anything else in the log, not about how that node got half lost, and not why the cluster went red. I wasn't able to reproduce that pending situation.  
It can't have been split brain, we have minimum\_master\_nodes = 2.  
Is it possible that the cluster didn't realize that it lost that node, tried to assign it a primary shard and went red because of this?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 1, 2015, 1:36am UTC](https://discuss.elastic.co/t/discovery-zen-publish-reports-pending-node-cluster-gets-red-at-index-create/24612/2 "2015-07-01T01:36:23Z")

</div>

Could be the node was overloaded, anything regarding GC in the logs before this happened?

---

<div class="post-metadata">

**Author:** ![Suny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suny/32/22082_2.png) [@Suny](https://discuss.elastic.co/u/Suny)\
**Post date:** [July 1, 2015, 1:08pm UTC](https://discuss.elastic.co/t/discovery-zen-publish-reports-pending-node-cluster-gets-red-at-index-create/24612/3 "2015-07-01T13:08:30Z")

</div>

It's a bit spooky, but there's no garbage collection log entry in the master's logs. These pending nodes occurred again some days later, but not for so long.  
For the unresponsive non-master nodes, their logfile simply end at the time when they have their first timeout. They do log garbage collections, but at other times.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:04am UTC](https://discuss.elastic.co/t/discovery-zen-publish-reports-pending-node-cluster-gets-red-at-index-create/24612/4 "2017-07-06T00:04:10Z")

</div>