Can't start ES 6.4 cluster with 3 master nodes

Elastic Stack Elasticsearch
1

Hello,
I have ES 6.3.2 cluster with 3 master nodes running with xpack.security.enabled=true
When I' trying to upgrade to ES 6.4.x, master nodes can't communicate each other and form
the cluster. Master's elasticsearch.yml is:
cluster.name: "bielk-xp"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 2
xpack.security.enabled: true
node.master: true
node.ingest:false
node.data: false
What has changed in the version 6.4.x for x-pack
ef
iintinos
Thanks? n advancie
d

2

You need to have at least one node in the cluster that is able to store data.

3

I have one data node in this environment but it also can't join the cluster:

[2018-10-08T06:07:10,720][INFO ][o.e.d.DiscoveryModule ] [es-data1-642] using discovery type [zen]
[2018-10-08T06:07:11,439][INFO ][o.e.n.Node ] [es-data1-642] initialized
[2018-10-08T06:07:11,440][INFO ][o.e.n.Node ] [es-data1-642] starting ...
[2018-10-08T06:07:11,585][INFO ][o.e.t.TransportService ] [es-data1-642] publish_address {10.244.4.126:9300}, bound_addresses {0.0.0.0:9300}
[2018-10-08T06:07:11,599][INFO ][o.e.b.BootstrapChecks ] [es-data1-642] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-10-08T06:07:14,878][INFO ][o.e.d.z.ZenDiscovery ] [es-data1-642] failed to send join request to master [{OjmeS4W}{OjmeS4WTRYGLHSV4ZRso7Q}{e3uOOIXGRiGqj27giMURsA}{10.244.5.203}{10.244.5.203:9300}{ml.machine_memory=270357331968, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[OjmeS4W][10.244.5.203:9300][internal:discovery/zen/join]]; nested: IllegalStateException[node {es-data1-642}{JgJuSL1BRYiYeXm1OrE3Jw}{k0EJVDJ6TTGwoh1_gAi3Jg}{10.244.4.126}{10.244.4.126:9300}{ml.machine_memory=270357331968, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} is on version [6.3.2] that cannot deserialize the license format [4], upgrade node to at least 6.4.0]; ]
[2018-10-08T06:07:17,946][INFO ][o.e.d.z.ZenDiscovery ] [es-data1-642] failed to send join request to master [{OjmeS4W}{OjmeS4WTRYGLHSV4ZRso7Q}{e3uOOIXGRiGqj27giMURsA}{10.244.5.203}{10.244.5.203:9300}{ml.machine_memory=270357331968, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[OjmeS4W][10.244.5.203:9300][internal:discovery/zen/join]]; nested: IllegalStateException[node {es-data1-642}{JgJuSL1BRYiYeXm1OrE3Jw}{k0EJVDJ6TTGwoh1_gAi3Jg}{10.244.4.126}{10.244.4.126:9300}{ml.machine_memory=270357331968, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} is on version [6.3.2] that cannot deserialize the license format [4], upgrade node to at least 6.4.0]; ]
[2018-10-08T06:07:20,948][WARN ][o.e.d.z.ZenDiscovery ] [es-data1-642] not enough master nodes discovered during pinging (found [[Candidate{node={sI_j8UQ}{sI_j8UQMTECgHjFXN9fszA}{uZI8EXDOS1249Knl2vyvEw}{10.244.6.250}{10.244.6.250:9300}{ml.machine_memory=270357331968, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-10-08T06:07:23,951][WARN ][o.e.d.z.ZenDiscovery ] [es-data1-642] not enough master nodes discovered during pinging (found [[Candidate{node={sI_j8UQ}{sI_j8UQMTECgHjFXN9fszA}{uZI8EXDOS1249Knl2vyvEw}{10.244.6.250}{10.244.6.250:9300}{ml.machine_memory=270357331968, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
,

4

The error you get indicates you need to upgrade Elasticsearch on your data node too, because it can't deserialize the license format and thus can't join the cluster.

5

By mistake started with prev. version. Now with 6.4.2 the same issue, datanode's log:

[2018-10-08T08:06:22,215][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/238] [Main.cc@109] controller (64 bit): Version 6.4.2 (Build 660eefe6f2ea55) Copyright (c) 2018 Elasticsearch BV
[2018-10-08T08:06:22,712][INFO ][o.e.d.DiscoveryModule ] [es-data1-642] using discovery type [zen]
[2018-10-08T08:06:23,549][INFO ][o.e.n.Node ] [es-data1-642] initialized
[2018-10-08T08:06:23,550][INFO ][o.e.n.Node ] [es-data1-642] starting ...
[2018-10-08T08:06:23,701][INFO ][o.e.t.TransportService ] [es-data1-642] publish_address {10.244.4.127:9300}, bound_addresses {0.0.0.0:9300}
[2018-10-08T08:06:23,717][INFO ][o.e.b.BootstrapChecks ] [es-data1-642] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-10-08T08:06:30,134][WARN ][o.e.d.z.ZenDiscovery ] [es-data1-642] not enough master nodes discovered during pinging (found [[Candidate{node={sI_j8UQ}{sI_j8UQMTECgHjFXN9fszA}{uZI8EXDOS1249Knl2vyvEw}{10.244.6.250}{10.244.6.250:9300}{ml.machine_memory=270357331968, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-10-08T08:06:33,137][WARN ][o.e.d.z.ZenDiscovery ] [es-data1-642] not enough master nodes discovered during pinging (found [[Candidate{node={sI_j8UQ}{sI_j8UQMTECgHjFXN9fszA}{uZI8EXDOS1249Knl2vyvEw}{10.244.6.250}{10.244.6.250:9300}{ml.machine_memory=270357331968, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-10-08T08:06:39,341][WARN ][o.e.d.z.ZenDiscovery ] [es-data1-642] not enough master nodes discovered during pinging (found [[Candidate{node={sI_j8UQ}{sI_j8UQMTECgHjFXN9fszA}{uZI8EXDOS1249Knl2vyvEw}{10.244.6.250}{10.244.6.250:9300}{ml.machine_memory=270357331968, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-10-08T08:06:42,344][WARN ][o.e.d.z.ZenDiscovery ] [es-data1-642] not enough master nodes discovered during pinging (found [[Candidate{node={sI_j8UQ}{sI_j8UQMTECgHjFXN9fszA}{uZI8EXDOS1249Knl2vyvEw}{10.244.6.250}{10.244.6.250:9300}{ml.machine_memory=270357331968, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-10-08T08:06:45,346][WARN ][o.e.d.z.ZenDiscovery ] [es-data1-642] not enough master nodes discovered during pinging (found [[Candidate{node={sI_j8UQ}{sI_j8UQMTECgHjFXN9fszA}{uZI8EXDOS1249Knl2vyvEw}{10.244.6.250}{10.244.6.250:9300}{ml.machine_memory=270357331968, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again

3 master nodes can't communicate with each other and not forming ES cluster
But with the same config and
.3.2 workin6gfine

6

The es-data1-642 data node seems to find only one master (the one at x.x.x.250) so for some reason the two other master candidate nodes have failed to join the cluster.

I'm not familiar with 6.4.2 so I can't tell if this is due to a change or not. What does the logfiles on the two other master candidates say when you restart them? Anything about failing to join the cluster? Do you use the same elasticsearch.yml file on all three master candidates or are there differences?

7

Thank you all. The issue resolved by activating trial 30-day x-pack license and Initiating the setup of passwords for reserved users elastic,kibana,logstash_system,beats_system.
After activation of the license, cluster successfully formed and functioning, what working without a trial license for ES 6.3.x

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.