# ES Cluster Health in yellow due to some Replica shards in Unassigned state

**URL:** <https://discuss.elastic.co/t/es-cluster-health-in-yellow-due-to-some-replica-shards-in-unassigned-state/157836>\
**Category:** Elasticsearch\
**Created:** [November 22, 2018, 9:13am UTC](https://discuss.elastic.co/t/es-cluster-health-in-yellow-due-to-some-replica-shards-in-unassigned-state/157836 "2018-11-22T09:13:07Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![damvinod](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/damvinod/32/53303_2.png) [@damvinod](https://discuss.elastic.co/u/damvinod)\
**Post date:** [November 22, 2018, 9:13am UTC](https://discuss.elastic.co/t/es-cluster-health-in-yellow-due-to-some-replica-shards-in-unassigned-state/157836/1 "2018-11-22T09:13:07Z")

</div>

We have elastic search Production cluster environment with 4 nodes

**Node 1:**  
node.master: true  
**Node 2:**  
node.master: true  
node.data: true  
**Node 3:**  
node.master: true  
node.data: true  
**Node 4: Coordinating node**  
node.master: false  
node.data: false  
node.ingest: false

We are facing an issue of replica shards getting allocated on the **Node 2**.  
No issues with the Primary shards in **Node 3** which is the current Master node

**Logs from Node 2 is as below:**

[2018-11-22T16:42:48,849][INFO][o.e.d.z.ZenDiscovery] [node-plmspapgs0g] master\_left [{node-plmspapgs0h}{ADnQ6UDZRoOFFvnkcdh3Sw}{R17y6k\_ZTyOTzIby9xlTIw}{X.X.X.X}}{X.X.X.X}:9300}{ml.machine\_memory=33559379968, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true}], reason [failed to ping, tried [3] times, each with maximum [30s] timeout]  
[2018-11-22T16:42:48,850][WARN][o.e.d.z.ZenDiscovery] [node-plmspapgs0g] master left (reason = failed to ping, tried [3] times, each with maximum [30s] timeout), current nodes: nodes:  
{node-plmspapgs0i}{31N0LIJiRtG3kfWsSoD2Iw}{RVg25n\_5TomjJpi307wBVQ}{X.X.X.X}{X.X.X.X}:9300}{ml.machine\_memory=16649068544, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true}  
{node-plmspapgs0e}{z49EVuIYQpOERYIYC2ZELA}{5WJEVl-NRtGH7A7qYUBxDg}{X.X.X.X}}{X.X.X.X}:9300}{ml.machine\_memory=33559379968, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true}  
{node-plmspapgs0h}{ADnQ6UDZRoOFFvnkcdh3Sw}{R17y6k\_ZTyOTzIby9xlTIw}{X.X.X.X}}{X.X.X.X}:9300}{ml.machine\_memory=33559379968, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true}, master  
{node-plmspapgs0g}{ffFONm7QQRqTkB8cH2DlVg}{wB9thnAdSE256v41YipqTA}{X.X.X.X}}{X.X.X.X}:9300}{ml.machine\_memory=33559379968, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}, local

[2018-11-22T16:42:50,007][WARN][r.suppressed] path: /\_xpack/monitoring/\_bulk, params: {system\_id=beats, system\_api\_version=6, interval=10s}  
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE\_UNAVAILABLE/2/no master];  
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:166) ~[elasticsearch-6.4.0.jar:6.4.0]  
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:152) ~[elasticsearch-6.4.0.jar:6.4.0]  
at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:56) ~[?:?]  
at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:36) ~[?:?]  
at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143) ~[elasticsearch-6.4.0.jar:6.4.0]  
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.4.0.jar:6.4.0]  
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:128) ~[?:?]

**As a workaround:**

After deleting all the indexes cluster health is in Green for the first 4-5 days.  
The cluster turned to Yellow because of replicas not being allocated to Node 2

Please try to help as we are facing this issue for very long time.

---

<div class="post-metadata">

**Author:** ![geekpete](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/geekpete/32/20409_2.png) [@geekpete](https://discuss.elastic.co/u/geekpete)\
**Post date:** [December 2, 2018, 11:25pm UTC](https://discuss.elastic.co/t/es-cluster-health-in-yellow-due-to-some-replica-shards-in-unassigned-state/157836/2 "2018-12-02T23:25:32Z")

</div>

What is your minimum masters setting?  
Is it the same on all nodes?

The logs indicate that Node 2 cannot find a master node, do you have any networking issues between Node 2 and the other nodes or generally?

Do you have any shard allocation awareness in play?  
[https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-awareness.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-awareness.html)

When you see the cluster go yellow can you provide an Allocation Explain output?  
GET /\_cluster/allocation/explain

What replica settings do you have on the indices that go yellow when losing 1 of 3 data nodes in the cluster?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 30, 2018, 11:25pm UTC](https://discuss.elastic.co/t/es-cluster-health-in-yellow-due-to-some-replica-shards-in-unassigned-state/157836/3 "2018-12-30T23:25:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.