# We have cluster of 4 nodes, where 2 nodes are master and data and other 2 nodes are data nodes, the configuration was working fine since 2 yrs, today we have to restart the cluster and since then we are getting master not discovered exception

**URL:** <https://discuss.elastic.co/t/we-have-cluster-of-4-nodes-where-2-nodes-are-master-and-data-and-other-2-nodes-are-data-nodes-the-configuration-was-working-fine-since-2-yrs-today-we-have-to-restart-the-cluster-and-since-then-we-are-getting-master-not-discovered-exception/340122>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-monitoring\
**Created:** [August 4, 2023, 10:50am UTC](https://discuss.elastic.co/t/we-have-cluster-of-4-nodes-where-2-nodes-are-master-and-data-and-other-2-nodes-are-data-nodes-the-configuration-was-working-fine-since-2-yrs-today-we-have-to-restart-the-cluster-and-since-then-we-are-getting-master-not-discovered-exception/340122 "2023-08-04T10:50:46Z")\
**Posts on this page:** 7\
**Page:** 2

<div class="post-metadata">

**Author:** ![vishnu\_ishpujani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishnu_ishpujani/32/123762_2.png) [@vishnu\_ishpujani](https://discuss.elastic.co/u/vishnu_ishpujani)\
**Post date:** [August 5, 2023, 11:59am UTC](https://discuss.elastic.co/t/we-have-cluster-of-4-nodes-where-2-nodes-are-master-and-data-and-other-2-nodes-are-data-nodes-the-configuration-was-working-fine-since-2-yrs-today-we-have-to-restart-the-cluster-and-since-then-we-are-getting-master-not-discovered-exception/340122/21 "2023-08-05T11:59:32Z")

</div>

@DavidTurner If you see this logs, these are the logs which I got without any change in the config.yml, from this logs can you tell where is the storage performance might be affecting the cluster?

---

<div class="post-metadata">

**Author:** ![vishnu\_ishpujani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishnu_ishpujani/32/123762_2.png) [@vishnu\_ishpujani](https://discuss.elastic.co/u/vishnu_ishpujani)\
**Post date:** [August 5, 2023, 12:05pm UTC](https://discuss.elastic.co/t/we-have-cluster-of-4-nodes-where-2-nodes-are-master-and-data-and-other-2-nodes-are-data-nodes-the-configuration-was-working-fine-since-2-yrs-today-we-have-to-restart-the-cluster-and-since-then-we-are-getting-master-not-discovered-exception/340122/22 "2023-08-05T12:05:37Z")

</div>

```auto
[2023-08-05T22:02:07,792][WARN][r.suppressed] [ES-Master-1] path: /_license, params: {human=false}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.16.2.jar:7.16.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]

```

```auto
[2023-08-05T22:02:58,576][WARN][r.suppressed] [ES-Master-2] path: /_license, params: {human=false}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.16.2.jar:7.16.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]

```

These are the logs that are getting printed by my old config, I don't see any storage performance issue. Sorry for asking this much questions, as I am new to elasticsearch

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 5, 2023, 12:17pm UTC](https://discuss.elastic.co/t/we-have-cluster-of-4-nodes-where-2-nodes-are-master-and-data-and-other-2-nodes-are-data-nodes-the-configuration-was-working-fine-since-2-yrs-today-we-have-to-restart-the-cluster-and-since-then-we-are-getting-master-not-discovered-exception/340122/23 "2023-08-05T12:17:49Z")

</div>

Those logs are not informative. The relevant log message is the one I highlighted above:

> [@We have cluster of 4 nodes, where 2 nodes are master and data and other 2 nodes are data nodes, the configuration was working fine since 2 yrs, today we have to restart the cluster and since then we are getting master not discovered exception](https://discuss.elastic.co/t/we-have-cluster-of-4-nodes-where-2-nodes-are-master-and-data-and-other-2-nodes-are-data-nodes-the-configuration-was-working-fine-since-2-yrs-today-we-have-to-restart-the-cluster-and-since-then-we-are-getting-master-not-discovered-exception/340122/8):
>
> The problem is here: your storage is so slow that it appears to be broken. You need to work out why it's so slow.

---

<div class="post-metadata">

**Author:** ![vishnu\_ishpujani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishnu_ishpujani/32/123762_2.png) [@vishnu\_ishpujani](https://discuss.elastic.co/u/vishnu_ishpujani)\
**Post date:** [August 6, 2023, 4:34am UTC](https://discuss.elastic.co/t/we-have-cluster-of-4-nodes-where-2-nodes-are-master-and-data-and-other-2-nodes-are-data-nodes-the-configuration-was-working-fine-since-2-yrs-today-we-have-to-restart-the-cluster-and-since-then-we-are-getting-master-not-discovered-exception/340122/24 "2023-08-06T04:34:01Z")

</div>

@DavidTurner currently we are using bursting throughput for EFS, can you recommend which throughput we should use so that it can resolve the issue?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 6, 2023, 7:30am UTC](https://discuss.elastic.co/t/we-have-cluster-of-4-nodes-where-2-nodes-are-master-and-data-and-other-2-nodes-are-data-nodes-the-configuration-was-working-fine-since-2-yrs-today-we-have-to-restart-the-cluster-and-since-then-we-are-getting-master-not-discovered-exception/340122/25 "2023-08-06T07:30:11Z")

</div>

I can't help with your EFS config, sorry. My best recommendation aligns with [the docs I shared above](https://www.elastic.co/guide/en/elasticsearch/plugins/current/cloud-aws-best-practices.html#_storage): move off EFS.

---

<div class="post-metadata">

**Author:** ![hlcxpl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hlcxpl/32/123290_2.png) [@hlcxpl](https://discuss.elastic.co/u/hlcxpl)\
**Post date:** [August 11, 2023, 2:41am UTC](https://discuss.elastic.co/t/we-have-cluster-of-4-nodes-where-2-nodes-are-master-and-data-and-other-2-nodes-are-data-nodes-the-configuration-was-working-fine-since-2-yrs-today-we-have-to-restart-the-cluster-and-since-then-we-are-getting-master-not-discovered-exception/340122/26 "2023-08-11T02:41:53Z")

</div>

you could change the storage or designe anotherone more fast chage your configuration to that path and remember to chanage the path in bought configuration files restar and try again of each nodes

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 8, 2023, 2:42am UTC](https://discuss.elastic.co/t/we-have-cluster-of-4-nodes-where-2-nodes-are-master-and-data-and-other-2-nodes-are-data-nodes-the-configuration-was-working-fine-since-2-yrs-today-we-have-to-restart-the-cluster-and-since-then-we-are-getting-master-not-discovered-exception/340122/27 "2023-09-08T02:42:06Z")

</div>

Closed on September 8, 2023, 2:42am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/we-have-cluster-of-4-nodes-where-2-nodes-are-master-and-data-and-other-2-nodes-are-data-nodes-the-configuration-was-working-fine-since-2-yrs-today-we-have-to-restart-the-cluster-and-since-then-we-are-getting-master-not-discovered-exception/340122.md?page=1)
