# Can't start ES 6.4 cluster with 3 master nodes
**URL:** https://discuss.elastic.co/t/cant-start-es-6-4-cluster-with-3-master-nodes/151348
**Category:** Elasticsearch
**Created:** [October 7, 2018, 11:58am UTC](https://discuss.elastic.co/t/cant-start-es-6-4-cluster-with-3-master-nodes/151348 "2018-10-07T11:58:56Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@ludaca](https://discuss.elastic.co/u/ludaca)
#### Post date: [October 7, 2018, 11:58am UTC](https://discuss.elastic.co/t/cant-start-es-6-4-cluster-with-3-master-nodes/151348/1 "2018-10-07T11:58:56Z")
Hello,
I have ES 6.3.2 cluster with 3 master nodes running with xpack.security.enabled=true
When I' trying to upgrade to ES 6.4.x, master nodes can't communicate each other and form
the cluster. Master's elasticsearch.yml is:
cluster.name: "bielk-xp"
network.host: 0.0.0.0
discovery.zen.minimum\_master\_nodes: 2
xpack.security.enabled: true
node.master: true
node.ingest:false
node.data: false
What has changed in the version 6.4.x for x-pack
ef
iintinos
Thanks? n advancie
d
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 7, 2018, 4:42pm UTC](https://discuss.elastic.co/t/cant-start-es-6-4-cluster-with-3-master-nodes/151348/2 "2018-10-07T16:42:22Z")
You need to have at least one node in the cluster that is able to store data.
---
### Author:  [@ludaca](https://discuss.elastic.co/u/ludaca)
#### Post date: [October 8, 2018, 6:13am UTC](https://discuss.elastic.co/t/cant-start-es-6-4-cluster-with-3-master-nodes/151348/3 "2018-10-08T06:13:33Z")
I have one data node in this environment but it also can't join the cluster:
[2018-10-08T06:07:10,720][INFO][o.e.d.DiscoveryModule] [es-data1-642] using discovery type [zen]
[2018-10-08T06:07:11,439][INFO][o.e.n.Node] [es-data1-642] initialized
[2018-10-08T06:07:11,440][INFO][o.e.n.Node] [es-data1-642] starting ...
[2018-10-08T06:07:11,585][INFO][o.e.t.TransportService] [es-data1-642] publish\_address {10.244.4.126:9300}, bound\_addresses {0.0.0.0:9300}
[2018-10-08T06:07:11,599][INFO][o.e.b.BootstrapChecks] [es-data1-642] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-10-08T06:07:14,878][INFO][o.e.d.z.ZenDiscovery] [es-data1-642] failed to send join request to master [{OjmeS4W}{OjmeS4WTRYGLHSV4ZRso7Q}{e3uOOIXGRiGqj27giMURsA}{10.244.5.203}{10.244.5.203:9300}{ml.machine\_memory=270357331968, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[OjmeS4W][10.244.5.203:9300][internal:discovery/zen/join]]; nested: IllegalStateException[node {es-data1-642}{JgJuSL1BRYiYeXm1OrE3Jw}{k0EJVDJ6TTGwoh1\_gAi3Jg}{10.244.4.126}{10.244.4.126:9300}{ml.machine\_memory=270357331968, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true} is on version [6.3.2] that cannot deserialize the license format [4], upgrade node to at least 6.4.0]; ]
[2018-10-08T06:07:17,946][INFO][o.e.d.z.ZenDiscovery] [es-data1-642] failed to send join request to master [{OjmeS4W}{OjmeS4WTRYGLHSV4ZRso7Q}{e3uOOIXGRiGqj27giMURsA}{10.244.5.203}{10.244.5.203:9300}{ml.machine\_memory=270357331968, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[OjmeS4W][10.244.5.203:9300][internal:discovery/zen/join]]; nested: IllegalStateException[node {es-data1-642}{JgJuSL1BRYiYeXm1OrE3Jw}{k0EJVDJ6TTGwoh1\_gAi3Jg}{10.244.4.126}{10.244.4.126:9300}{ml.machine\_memory=270357331968, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true} is on version [6.3.2] that cannot deserialize the license format [4], upgrade node to at least 6.4.0]; ]
[2018-10-08T06:07:20,948][WARN][o.e.d.z.ZenDiscovery] [es-data1-642] not enough master nodes discovered during pinging (found [[Candidate{node={sI\_j8UQ}{sI\_j8UQMTECgHjFXN9fszA}{uZI8EXDOS1249Knl2vyvEw}{10.244.6.250}{10.244.6.250:9300}{ml.machine\_memory=270357331968, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-10-08T06:07:23,951][WARN][o.e.d.z.ZenDiscovery] [es-data1-642] not enough master nodes discovered during pinging (found [[Candidate{node={sI\_j8UQ}{sI\_j8UQMTECgHjFXN9fszA}{uZI8EXDOS1249Knl2vyvEw}{10.244.6.250}{10.244.6.250:9300}{ml.machine\_memory=270357331968, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
,
---
### Author:  [@Bernt\_Rostad](https://discuss.elastic.co/u/Bernt_Rostad)
#### Post date: [October 8, 2018, 7:49am UTC](https://discuss.elastic.co/t/cant-start-es-6-4-cluster-with-3-master-nodes/151348/4 "2018-10-08T07:49:39Z")
> [@ludaca](#):
>
> is on version [6.3.2] that cannot deserialize the license format [4], upgrade node to at least 6.4.0]; ]
The error you get indicates you need to upgrade Elasticsearch on your data node too, because it can't deserialize the license format and thus can't join the cluster.
---
### Author:  [@ludaca](https://discuss.elastic.co/u/ludaca)
#### Post date: [October 8, 2018, 8:11am UTC](https://discuss.elastic.co/t/cant-start-es-6-4-cluster-with-3-master-nodes/151348/5 "2018-10-08T08:11:57Z")
By mistake started with prev. version. Now with 6.4.2 the same issue, datanode's log:
[2018-10-08T08:06:22,215][INFO][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/238] [Main.cc@109] controller (64 bit): Version 6.4.2 (Build 660eefe6f2ea55) Copyright (c) 2018 Elasticsearch BV
[2018-10-08T08:06:22,712][INFO][o.e.d.DiscoveryModule] [es-data1-642] using discovery type [zen]
[2018-10-08T08:06:23,549][INFO][o.e.n.Node] [es-data1-642] initialized
[2018-10-08T08:06:23,550][INFO][o.e.n.Node] [es-data1-642] starting ...
[2018-10-08T08:06:23,701][INFO][o.e.t.TransportService] [es-data1-642] publish\_address {10.244.4.127:9300}, bound\_addresses {0.0.0.0:9300}
[2018-10-08T08:06:23,717][INFO][o.e.b.BootstrapChecks] [es-data1-642] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-10-08T08:06:30,134][WARN][o.e.d.z.ZenDiscovery] [es-data1-642] not enough master nodes discovered during pinging (found [[Candidate{node={sI\_j8UQ}{sI\_j8UQMTECgHjFXN9fszA}{uZI8EXDOS1249Knl2vyvEw}{10.244.6.250}{10.244.6.250:9300}{ml.machine\_memory=270357331968, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-10-08T08:06:33,137][WARN][o.e.d.z.ZenDiscovery] [es-data1-642] not enough master nodes discovered during pinging (found [[Candidate{node={sI\_j8UQ}{sI\_j8UQMTECgHjFXN9fszA}{uZI8EXDOS1249Knl2vyvEw}{10.244.6.250}{10.244.6.250:9300}{ml.machine\_memory=270357331968, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-10-08T08:06:39,341][WARN][o.e.d.z.ZenDiscovery] [es-data1-642] not enough master nodes discovered during pinging (found [[Candidate{node={sI\_j8UQ}{sI\_j8UQMTECgHjFXN9fszA}{uZI8EXDOS1249Knl2vyvEw}{10.244.6.250}{10.244.6.250:9300}{ml.machine\_memory=270357331968, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-10-08T08:06:42,344][WARN][o.e.d.z.ZenDiscovery] [es-data1-642] not enough master nodes discovered during pinging (found [[Candidate{node={sI\_j8UQ}{sI\_j8UQMTECgHjFXN9fszA}{uZI8EXDOS1249Knl2vyvEw}{10.244.6.250}{10.244.6.250:9300}{ml.machine\_memory=270357331968, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-10-08T08:06:45,346][WARN][o.e.d.z.ZenDiscovery] [es-data1-642] not enough master nodes discovered during pinging (found [[Candidate{node={sI\_j8UQ}{sI\_j8UQMTECgHjFXN9fszA}{uZI8EXDOS1249Knl2vyvEw}{10.244.6.250}{10.244.6.250:9300}{ml.machine\_memory=270357331968, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
3 master nodes can't communicate with each other and not forming ES cluster
But with the same config and
.3.2 workin6gfine
---
### Author:  [@Bernt\_Rostad](https://discuss.elastic.co/u/Bernt_Rostad)
#### Post date: [October 8, 2018, 9:31am UTC](https://discuss.elastic.co/t/cant-start-es-6-4-cluster-with-3-master-nodes/151348/6 "2018-10-08T09:31:12Z")
> [@ludaca](#):
>
> [es-data1-642] not enough master nodes discovered during pinging (found [[Candidate{node={sI\_j8UQ}{sI\_j8UQMTECgHjFXN9fszA}{uZI8EXDOS1249Knl2vyvEw}{10.244.6.250}{10.244.6.250:9300}{ml.machine\_memory=270357331968, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
The es-data1-642 data node seems to find only one master (the one at x.x.x.250) so for some reason the two other master candidate nodes have failed to join the cluster.
I'm not familiar with 6.4.2 so I can't tell if this is due to a change or not. What does the logfiles on the two other master candidates say when you restart them? Anything about failing to join the cluster? Do you use the same elasticsearch.yml file on all three master candidates or are there differences?
---
### Author:  [@ludaca](https://discuss.elastic.co/u/ludaca)
#### Post date: [October 9, 2018, 7:38am UTC](https://discuss.elastic.co/t/cant-start-es-6-4-cluster-with-3-master-nodes/151348/7 "2018-10-09T07:38:57Z")
Thank you all. The issue resolved by activating trial 30-day x-pack license and Initiating the setup of passwords for reserved users elastic,kibana,logstash\_system,beats\_system.
After activation of the license, cluster successfully formed and functioning, what working without a trial license for ES 6.3.x
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 6, 2018, 7:38am UTC](https://discuss.elastic.co/t/cant-start-es-6-4-cluster-with-3-master-nodes/151348/8 "2018-11-06T07:38:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.