# Getting "master not discovered or elected yet" causing cluster not up in version 7.1.0

**URL:** <https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489>\
**Category:** Elasticsearch\
**Created:** [May 23, 2019, 4:22pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489 "2019-05-23T16:22:29Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![niudaye123](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@niudaye123](https://discuss.elastic.co/u/niudaye123)\
**Post date:** [May 23, 2019, 4:22pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/1 "2019-05-23T16:22:29Z")

</div>

Hi, I am building a 6 nodes cluster, node 1-3 master nodes, and node 4-6 data nodes. I have the following in elasticsearch.yml on each node:

bootstrap.memory\_lock: false  
cluster.initial\_master\_nodes:

- awselsdevlap01.est1933.com-esnode01
- awselsdevlap02.est1933.com-esnode02
- awselsdevlap03.est1933.com-esnode03  
cluster.name: es\_gallo\_dev  
discovery.zen.minimum\_master\_nodes: 3  
discovery.zen.ping.unicast.hosts: [awselsdevlap01.est1933.com](http://awselsdevlap01.est1933.com), [awselsdevlap02.est1933.com](http://awselsdevlap02.est1933.com),  
[awselsdevlap03.est1933.com](http://awselsdevlap03.est1933.com), [awselsdevlap04.est1933.com](http://awselsdevlap04.est1933.com), [awselsdevlap05.est1933.com](http://awselsdevlap05.est1933.com),  
[awselsdevlap06.est1933.com](http://awselsdevlap06.est1933.com)  
http.port: 9200  
network.host: _site_  
transport.tcp.port: 9300  
xpack.security.authc.realms.ldap.ldap1.bind\_dn: uid=s-elasticsearch,ou=people,[o=ejgallo.com](http://o=ejgallo.com)  
xpack.security.authc.realms.ldap.ldap1.group\_search.base\_dn: ou=groups,[o=ejgallo.com](http://o=ejgallo.com)  
xpack.security.authc.realms.ldap.ldap1.order: 1  
xpack.security.authc.realms.ldap.ldap1.url: ldaps://gdsprd01.ejgallo.com:636  
xpack.security.authc.realms.ldap.ldap1.user\_search.base\_dn: ou=people,[o=ejgallo.com](http://o=ejgallo.com)  
xpack.security.enabled: true

node.name: awselsdevlap01.est1933.com-esnode01

#################################### Paths ####################################

# Path to directory containing configuration (this file and logging.yml):

path.data: /es\_data/data01/awselsdevlap01.est1933.com-esnode01,/es\_data/data02/awselsdevlap01.est1933.com-esnode01,/es\_data/data03/awselsdevlap01.est1933.com-esnode01,/es\_data/data04/awselsdevlap01.est1933.com-esnode01,/es\_data/data05/awselsdevlap01.est1933.com-esnode01

path.logs: /es\_data/es\_logs/awselsdevlap01.est1933.com-esnode01

The following is what the logs look like:

2019-05-23T00:00:35,579][WARN][o.e.c.c.ClusterFormationFailureHelper] [awselsdevlap01.est1933.com-esnode01] master not discovered or elected yet, an election requires at least 2 nodes with ids from [UWgnBPsHQ1aW4xXEZVKyJQ, T\_8EuKpaTqWg2oP3TAnAaA, YZ6m2ioDQWqi1cNnOteB6w], have discovered [{awselsdevlap02.est1933.com-esnode02}{T\_8EuKpaTqWg2oP3TAnAaA}{-D-VHjdeSUyJdlTauLVuQw}{10.173.148.65}{10.173.148.65:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}, {awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{-CoUrjn9QlKE-K5SqZ-JYw}{10.173.148.73}{10.173.148.73:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300] from hosts providers and [{awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{epXUK1dTSKCf0Ca9CphE3A}{10.173.148.143}{10.173.148.143:9300}{ml.machine\_memory=31980478464, xpack.installed=true, ml.max\_open\_jobs=20}] from last-known cluster state; node term 22, last-accepted version 2867 in term 21

Any idea what wrong with my config? I checked other similar postings and ensured the list in cluster.initial\_master\_nodes match the node.name setting.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 23, 2019, 4:28pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/2 "2019-05-23T16:28:23Z")

</div>

> [@niudaye123](#):
>
> have discovered ... which is a quorum

Hmm, this means the nodes have all found each other but for some reason cannot form a cluster. Are there any other log messages? Note that there's an open issue to do with insufficient logging if your security configuration is broken - see [#42153](https://github.com/elastic/elasticsearch/issues/42153). Can you try setting:

```auto
logger.org.elasticsearch.discovery: TRACE

```

and see if you get any more useful messages?

---

<div class="post-metadata">

**Author:** ![niudaye123](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@niudaye123](https://discuss.elastic.co/u/niudaye123)\
**Post date:** [May 23, 2019, 5:09pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/3 "2019-05-23T17:09:20Z")

</div>

I turned on the trace, I see bunch errors about connection refused. Not sure why? Even to itself.

[2019-05-23T16:45:29,789][DEBUG][o.e.d.PeerFinder] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed  
org.elasticsearch.transport.ConnectTransportException: [10.173.148.58:9300] connect\_exception  
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]  
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]  
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]  
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]  
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]  
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]  
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]  
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]  
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]  
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]  
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]  
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]  
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]  
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]  
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]  
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]  
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]  
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]  
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]  
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]  
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]  
at java.lang.Thread.run(Thread.java:835) [?:?]  
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: [awselsdevlap04.est1933.com/10.173.148.58:9300](http://awselsdevlap04.est1933.com/10.173.148.58:9300)  
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]  
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]  
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]  
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]  
... 6 more  
Caused by: java.net.ConnectException: Connection refused  
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]  
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]  
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]  
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]  
... 6 more

full log is here. [https://pastebin.com/kY7FghPU](https://pastebin.com/kY7FghPU)

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 23, 2019, 5:55pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/4 "2019-05-23T17:55:40Z")

</div>

```auto
discovery.zen.minimum_master_nodes: 3
discovery.zen.ping.unicast.hosts: awselsdevlap01.est1933.com, awselsdevlap02.est1933.com, awselsdevlap03.est1933.com, awselsdevlap04.est1933.com, awselsdevlap05.est1933.com, awselsdevlap06.est1933.com

```

In 7.1.0 you don't need `discovery.zen.minimum_master_nodes` and `discovery.zen.ping.unicast.hosts` is now called `discovery.seed_hosts`.

Also `discovery.seed_hosts` should only really contain the master-eligible nodes and I think that is the source of most of the `Connection refused` messages.

You also don't need `cluster.initial_master_nodes` any more since this cluster already started up at least once.

However none of that actually answers the question of why this cluster isn't forming.

Can you try fixing these things above and then setting the following?

```auto
logger.org.elasticsearch.cluster.coordination: TRACE

```

---

<div class="post-metadata">

**Author:** ![niudaye123](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@niudaye123](https://discuss.elastic.co/u/niudaye123)\
**Post date:** [May 23, 2019, 7:02pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/5 "2019-05-23T19:02:31Z")

</div>

I updated the yml file as below,

bootstrap.memory\_lock: false  
discovery.seed\_hosts: [awselsdevlap01.est1933.com](http://awselsdevlap01.est1933.com), [awselsdevlap02.est1933.com](http://awselsdevlap02.est1933.com),  
[awselsdevlap03.est1933.com](http://awselsdevlap03.est1933.com)  
http.port: 9200  
network.host: _site_  
transport.tcp.port: 9300  
xpack.security.authc.realms.ldap.ldap1.bind\_dn: uid=s-elasticsearch,ou=people,[o=ejgallo.com](http://o=ejgallo.com)  
xpack.security.authc.realms.ldap.ldap1.group\_search.base\_dn: ou=groups,[o=ejgallo.com](http://o=ejgallo.com)  
xpack.security.authc.realms.ldap.ldap1.order: 1  
xpack.security.authc.realms.ldap.ldap1.url: ldaps://gdsprd01.ejgallo.com:636  
xpack.security.authc.realms.ldap.ldap1.user\_search.base\_dn: ou=people,[o=ejgallo.com](http://o=ejgallo.com)  
xpack.security.enabled: true

node.name: awselsdevlap01.est1933.com-esnode01

#################################### Paths ####################################

# Path to directory containing configuration (this file and logging.yml):

path.data: /es\_data/data01/awselsdevlap01.est1933.com-esnode01,/es\_data/data02/awselsdevlap01.est1933.com-esnode01,/es\_data/data03/awselsdevlap01.est1933.com-esnode01,/es\_data/data04/awselsdevlap01.est1933.com-esnode01,/es\_data/data05/awselsdevlap01.est1933.com-esnode01

path.logs: /es\_data/es\_logs/awselsdevlap01.est1933.com-esnode01

action.auto\_create\_index: true

logger.org.elasticsearch.discovery: TRACE  
logger.org.elasticsearch.cluster.coordination: TRACE

Now I am getting error in the logs, errors are different now, but still no quorum is formed.

[https://pastebin.com/bXvj9D1v](https://pastebin.com/bXvj9D1v)

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 23, 2019, 7:31pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/6 "2019-05-23T19:31:59Z")

</div>

Thanks, this seems to be indicating that `awselsdevlap03.est1933.com-esnode03` should be winning the election. Could you enable the same logging on that node and share the logs from it too?

---

<div class="post-metadata">

**Author:** ![niudaye123](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@niudaye123](https://discuss.elastic.co/u/niudaye123)\
**Post date:** [May 23, 2019, 8:17pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/7 "2019-05-23T20:17:07Z")

</div>

Did that. here is the log from 03.

[https://pastebin.com/XNE4MUsV](https://pastebin.com/XNE4MUsV)

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 23, 2019, 8:50pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/8 "2019-05-23T20:50:11Z")

</div>

Ok that explains it:

```nohighlight
[2019-05-23T20:10:42,549][WARN][o.e.c.c.ClusterFormationFailureHelper] [awselsdevlap03.est1933.com-esnode03] master not discovered or elected yet, an election requires at least 3 nodes with ids from [UWgnBPsHQ1aW4xXEZVKyJQ, CM68Qk8DQ3KMZDDPc7wHkw, s3s5fQG6SwyWvtDPDKZ2gQ, 5emh3C9PQ_aK9Si-3uF_iQ, YZ6m2ioDQWqi1cNnOteB6w], have discovered [{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}] which is not a quorum; discovery will continue using [10.173.148.143:9300, 10.173.148.65:9300] from hosts providers and [{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{zQJRt6NJTS6ysTIo3VIbqg}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 22, last-accepted version 120434 in term 22

```

Here's what's needed for a successful election:

`an election requires at least 3 nodes with ids from [UWgnBPsHQ1aW4xXEZVKyJQ, CM68Qk8DQ3KMZDDPc7wHkw, s3s5fQG6SwyWvtDPDKZ2gQ, 5emh3C9PQ_aK9Si-3uF_iQ, YZ6m2ioDQWqi1cNnOteB6w]`

One of those node IDs is `awselsdevlap03`:

```auto
{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}

```

Another is `awselsdevlap01`:

```auto
{awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}

```

But `awselsdevlap02` isn't in that list:

```auto
{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}

```

I think this means that you have removed half or more of the master-eligible nodes in the cluster, so none of the remaining nodes has an up-to-date cluster state. You need to reinstate at least one of the missing nodes to proceed.

Can you tell us a bit more about what you've been doing to this cluster? Were you running it with a different configuration in the past? Were you using any unusual settings?

---

<div class="post-metadata">

**Author:** ![niudaye123](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@niudaye123](https://discuss.elastic.co/u/niudaye123)\
**Post date:** [May 23, 2019, 9:21pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/9 "2019-05-23T21:21:28Z")

</div>

I did the initial installs using the Ansible script from [https://github.com/elastic/ansible-elasticsearch](https://github.com/elastic/ansible-elasticsearch), which by default the security is not enabled.

I was able to get the cluster running after that, then I need to get LDAP integration to work, which needs to enable the security. So I added the lines for the LDAP realm, which was working in version 6.7.0. But once that is added, I never was able to bring up cluster again.

So at this point, what should I do to get cluster going without pulling all my hairs out? ☹

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 23, 2019, 9:49pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/10 "2019-05-23T21:49:43Z")

</div>

> [@niudaye123](#):
>
> So at this point, what should I do to get cluster going without pulling all my hairs out? ☹

As I said, you need to reinstate at least one of the missing nodes to proceed.

---

<div class="post-metadata">

**Author:** ![niudaye123](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@niudaye123](https://discuss.elastic.co/u/niudaye123)\
**Post date:** [May 23, 2019, 9:59pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/11 "2019-05-23T21:59:36Z")

</div>

I made the same change in yml file on the node 02, then logs from 01, 02 and 03 are all showing such as below:

[2019-05-23T21:43:08,343][TRACE][o.e.d.SeedHostsResolver] [awselsdevlap02.est1933.com-esnode02] resolved host [[awselsdevlap01.est1933.com](http://awselsdevlap01.est1933.com)] to [10.173.148.143:9300]  
[2019-05-23T21:43:08,343][TRACE][o.e.d.SeedHostsResolver] [awselsdevlap02.est1933.com-esnode02] resolved host [[awselsdevlap02.est1933.com](http://awselsdevlap02.est1933.com)] to [10.173.148.65:9300]  
[2019-05-23T21:43:08,343][TRACE][o.e.d.SeedHostsResolver] [awselsdevlap02.est1933.com-esnode02] resolved host [[awselsdevlap03.est1933.com](http://awselsdevlap03.est1933.com)] to [10.173.148.73:9300]  
[2019-05-23T21:43:08,343][TRACE][o.e.d.PeerFinder] [awselsdevlap02.est1933.com-esnode02] probing resolved transport addresses [10.173.148.143:9300, 10.173.148.73:9300]  
[2019-05-23T21:43:08,343][TRACE][o.e.d.PeerFinder] [awselsdevlap02.est1933.com-esnode02] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{zA7D9P01SaalWMy7pn5YdA}{10.173.148.73}{10.173.148.73:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{-J7niYWCS--b6Va3HjvSCQ}{10.173.148.143}{10.173.148.143:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}, {awselsdevlap02.est1933.com-esnode02}{T\_8EuKpaTqWg2oP3TAnAaA}{WYtQnncWTj26CKXTjDeDYA}{10.173.148.65}{10.173.148.65:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}], term=22}  
[2019-05-23T21:43:08,343][TRACE][o.e.d.PeerFinder] [awselsdevlap02.est1933.com-esnode02] startProbe(10.173.148.65:9300) not probing local node  
[2019-05-23T21:43:08,537][TRACE][o.e.d.PeerFinder] [awselsdevlap02.est1933.com-esnode02] startProbe(10.173.148.65:9300) not probing local node  
[2019-05-23T21:43:08,705][TRACE][o.e.d.PeerFinder] [awselsdevlap02.est1933.com-esnode02] startProbe(10.173.148.65:9300) not probing local node  
[2019-05-23T21:43:09,343][TRACE][o.e.d.PeerFinder] [awselsdevlap02.est1933.com-esnode02] Peer{transportAddress=10.173.148.143:9300, discoveryNode={awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{-J7niYWCS--b6Va3HjvSCQ}{10.173.148.143}{10.173.148.143:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers  
[2019-05-23T21:43:09,343][TRACE][o.e.d.PeerFinder] [awselsdevlap02.est1933.com-esnode02] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{zA7D9P01SaalWMy7pn5YdA}{10.173.148.73}{10.173.148.73:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers  
[2019-05-23T21:43:09,343][TRACE][o.e.d.PeerFinder] [awselsdevlap02.est1933.com-esnode02] probing master nodes from cluster state: nodes:  
{awselsdevlap02.est1933.com-esnode02}{T\_8EuKpaTqWg2oP3TAnAaA}{WYtQnncWTj26CKXTjDeDYA}{10.173.148.65}{10.173.148.65:9300}{ml.machine\_memory=31980478464, xpack.installed=true, ml.max\_open\_jobs=20}, local

[2019-05-23T21:43:09,343][TRACE][o.e.d.PeerFinder] [awselsdevlap02.est1933.com-esnode02] startProbe(10.173.148.65:9300) not probing local node  
[2019-05-23T21:43:09,344][TRACE][o.e.d.PeerFinder] [awselsdevlap02.est1933.com-esnode02] Peer{transportAddress=10.173.148.143:9300, discoveryNode={awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{-J7niYWCS--b6Va3HjvSCQ}{10.173.148.143}{10.173.148.143:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{zA7D9P01SaalWMy7pn5YdA}{10.173.148.73}{10.173.148.73:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}, {awselsdevlap02.est1933.com-esnode02}{T\_8EuKpaTqWg2oP3TAnAaA}{WYtQnncWTj26CKXTjDeDYA}{10.173.148.65}{10.173.148.65:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}], term=22}  
[2019-05-23T21:43:09,344][TRACE][o.e.d.PeerFinder] [awselsdevlap02.est1933.com-esnode02] startProbe(10.173.148.65:9300) not probing local node  
[2019-05-23T21:43:09,344][TRACE][o.e.d.SeedHostsResolver] [awselsdevlap02.est1933.com-esnode02] resolved host [[awselsdevlap01.est1933.com](http://awselsdevlap01.est1933.com)] to [10.173.148.143:9300]  
[2019-05-23T21:43:09,344][TRACE][o.e.d.SeedHostsResolver] [awselsdevlap02.est1933.com-esnode02] resolved host [[awselsdevlap02.est1933.com](http://awselsdevlap02.est1933.com)] to [10.173.148.65:9300]

---

<div class="post-metadata">

**Author:** ![niudaye123](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@niudaye123](https://discuss.elastic.co/u/niudaye123)\
**Post date:** [May 28, 2019, 4:43pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/12 "2019-05-28T16:43:05Z")

</div>

Hi, I am still not able to form a cluster, what else could not incorrect?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 28, 2019, 4:59pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/13 "2019-05-28T16:59:07Z")

</div>

What actions have you taken since your last post? What are the logs saying now?

---

<div class="post-metadata">

**Author:** ![niudaye123](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@niudaye123](https://discuss.elastic.co/u/niudaye123)\
**Post date:** [May 28, 2019, 5:14pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/14 "2019-05-28T17:14:37Z")

</div>

In each node, it listed the node to be resolved, and looks like it was able to find the other peer nodes, but still cluster is not able to formed.

[2019-05-28T17:07:59,191][TRACE][o.e.d.PeerFinder] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node  
[2019-05-28T17:07:59,191][TRACE][o.e.d.SeedHostsResolver] [awselsdevlap01.est1933.com-esnode01] resolved host [[awselsdevlap01.est1933.com](http://awselsdevlap01.est1933.com)] to [10.173.148.143:9300]  
[2019-05-28T17:07:59,191][TRACE][o.e.d.SeedHostsResolver] [awselsdevlap01.est1933.com-esnode01] resolved host [[awselsdevlap02.est1933.com](http://awselsdevlap02.est1933.com)] to [10.173.148.65:9300]  
[2019-05-28T17:07:59,191][TRACE][o.e.d.SeedHostsResolver] [awselsdevlap01.est1933.com-esnode01] resolved host [[awselsdevlap03.est1933.com](http://awselsdevlap03.est1933.com)] to [10.173.148.73:9300]  
[2019-05-28T17:07:59,191][TRACE][o.e.d.PeerFinder] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300]  
[2019-05-28T17:07:59,192][TRACE][o.e.d.PeerFinder] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T\_8EuKpaTqWg2oP3TAnAaA}{9j1XEYckRWaCHQDkymoHtw}{10.173.148.65}{10.173.148.65:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{-SSC-sLSShuU\_Zh-6sOScA}{10.173.148.143}{10.173.148.143:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}, {awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{NGvD66xeQyWTQ8u4jwnWnQ}{10.173.148.73}{10.173.148.73:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}], term=22}  
[2019-05-28T17:07:59,192][TRACE][o.e.d.PeerFinder] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node  
[2019-05-28T17:07:59,192][TRACE][o.e.d.PeerFinder] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{NGvD66xeQyWTQ8u4jwnWnQ}{10.173.148.73}{10.173.148.73:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{-SSC-sLSShuU\_Zh-6sOScA}{10.173.148.143}{10.173.148.143:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}, {awselsdevlap02.est1933.com-esnode02}{T\_8EuKpaTqWg2oP3TAnAaA}{9j1XEYckRWaCHQDkymoHtw}{10.173.148.65}{10.173.148.65:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}], term=22}  
[2019-05-28T17:07:59,192][TRACE][o.e.d.PeerFinder] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node  
[2019-05-28T17:07:59,481][TRACE][o.e.d.PeerFinder] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node  
[2019-05-28T17:07:59,655][TRACE][o.e.d.PeerFinder] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node  
[2019-05-28T17:08:00,191][TRACE][o.e.d.PeerFinder] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T\_8EuKpaTqWg2oP3TAnAaA}{9j1XEYckRWaCHQDkymoHtw}{10.173.148.65}{10.173.148.65:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers  
[2019-05-28T17:08:00,191][TRACE][o.e.d.PeerFinder] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{NGvD66xeQyWTQ8u4jwnWnQ}{10.173.148.73}{10.173.148.73:9300}{ml.machine\_memory=31980478464, ml.max\_open\_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers  
[2019-05-28T17:08:00,192][TRACE][o.e.d.PeerFinder] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:  
{awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{-SSC-sLSShuU\_Zh-6sOScA}{10.173.148.143}{10.173.148.143:9300}{ml.machine\_memory=31980478464, xpack.installed=true, ml.max\_open\_jobs=20}, local

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 28, 2019, 5:17pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/15 "2019-05-28T17:17:40Z")

</div>

It does not look like you have reinstated any of the missing nodes as instructed above:

> [@DavidTurner](#):
>
> As I said, you need to reinstate at least one of the missing nodes to proceed.

---

<div class="post-metadata">

**Author:** ![niudaye123](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@niudaye123](https://discuss.elastic.co/u/niudaye123)\
**Post date:** [May 28, 2019, 5:24pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/16 "2019-05-28T17:24:25Z")

</div>

When you say re-instate a node, what exactly that means? Excuse my lack of Elasticsearch terms. I have 6 nodes, 3 master nodes and 3 data nodes. I started all 3 master nodes. The log entries in each node are almost the same.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 28, 2019, 5:29pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/17 "2019-05-28T17:29:17Z")

</div>

When this cluster was last running there were more master-eligible nodes than there are now. The last log you shared was saying this:

> [@DavidTurner](#):
>
> `an election requires at least 3 nodes with ids from [UWgnBPsHQ1aW4xXEZVKyJQ, CM68Qk8DQ3KMZDDPc7wHkw, s3s5fQG6SwyWvtDPDKZ2gQ, 5emh3C9PQ_aK9Si-3uF_iQ, YZ6m2ioDQWqi1cNnOteB6w]`

At the moment, only 2 of the 5 listed nodes are available, and this is preventing an election from taking place. In order to allow a master to be elected you need to put one of the missing master-eligible nodes back into service. Unfortunately we can't really tell where these nodes have gone - all we know is that they're not running and talking to the current set of nodes.

---

<div class="post-metadata">

**Author:** ![Corneliu](https://avatars.discourse-cdn.com/v4/letter/c/9f8e36/32.png) [@Corneliu](https://discuss.elastic.co/u/Corneliu)\
**Post date:** [May 29, 2019, 6:14pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/18 "2019-05-29T18:14:31Z")

</div>

Try setting the following into elasticsearch.yml config file on all master nodes:

```auto
cluster.initial_master_nodes: ["node_name1", "node_name2", ...]

```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 29, 2019, 7:15pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/19 "2019-05-29T19:15:31Z")

</div>

> [@Corneliu](#):
>
> Try setting the following into elasticsearch.yml config file on all master nodes:

Hi @Corneliu, this won't help. The `cluster.initial_master_nodes` setting is ignored once the first election has completed, which is the case in this cluster.

---

<div class="post-metadata">

**Author:** ![niudaye123](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@niudaye123](https://discuss.elastic.co/u/niudaye123)\
**Post date:** [May 29, 2019, 11:02pm UTC](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489/20 "2019-05-29T23:02:50Z")

</div>

I am still fighting with this cluster not able to form issue, I do have a related question. Where the last cluster state is saved at? I want to wipe out the history of cluster and start over from scratch.

[Next page](https://discuss.elastic.co/t/getting-master-not-discovered-or-elected-yet-causing-cluster-not-up-in-version-7-1-0/182489.md?page=2)