Hi,
I maybe fail to understand the principles of zen discovery. But I can not make the Elasticsearch nodes join the cluster.
Info:
- All Elasticsearch nodes live in Docker containers
- Everything happens on my local machine for testing purpose
Now I want 2 master-only nodes to form a cluster. Here the relevant part of the config:
network.bind_host: 0.0.0.0
network.publish_host: 10.50.15.26 <– my local ip address
discovery.zen.minimum_master_nodes: 2
The first two lines are important for nodes inside Docker containers to communicate with each other and are less important for Zen Discovery I guess.
Without configuring transport.tcp.port and discovery.zen.ping.unicast.hosts the nodes fail to start.
This is how I start the containers:
docker run --name master1 -p 9300:9300 <elasticsearch_image>
docker run --name master2 -p 9301:9301 <elasticsearch_image>
This what I got:
[2019-01-29T14:40:15,370][WARN][o.e.d.z.ZenDiscovery] [master1]
not enough master nodes discovered during pinging
(found [[Candidate{node={master1}{fLG4b6OESXqlRPZMlU8Jyg}
{nBpqu7D9SzSme3-nD4JX3A}{10.50.15.26}{10.50.15.26:9300}
{ml.machine_memory=16694579200, xpack.installed=true, ml.max_open_jobs=20,
ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
It seems as if master1 found itself and stopped looking for other nodes. Same for master2.
Now I do some configuration like this:
master1:
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: 10.50.15.26:9301
master2:
transport.tcp.port: 9301
discovery.zen.ping.unicast.hosts: 10.50.15.26:9300
With this config everything runs fine.
–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
While it can be reasonable to stick with this configuration for the master-only nodes (as they tend to stay in threes) it only shifts the problem.
Let's say I want to a bunch of data nodes or coordinating-only nodes. The configuration looks like this:
network.bind_host: 0.0.0.0
network.publish_host: 10.50.15.26
http.port: 9210-9219
transport.tcp.port: 9310-9319
discovery.zen:
minimum_master_nodes: 2
ping.unicast.hosts:
- 10.50.15.26:9300 <– master1
- 10.50.15.26:9301 <– master2
- 10.50.15.26:9302 <– master3
Start them:
docker run --name data1 -p 9210:9210 -p 9310:9310 <elasticsearch_image>
docker run --name data2 -p 9211:9211 -p 9311:9311 <elasticsearch_image>
...
The first one successfully joins the cluster! The second gives this error:
[2019-01-29T14:46:29,157][INFO ][o.e.d.z.ZenDiscovery] [data2]
failed to send join request to master [{master2}{1bhY71VgS_6WIhg1Xu6ZIw}
{NyojrkngTDamazzfeDfHjg}{10.50.15.26}{10.50.15.26:9301}
{ml.machine_memory=16694579200, ml.max_open_jobs=20,
xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException
[[master2][172.24.0.4:9301][internal:discovery/zen/join]]; nested:
ConnectTransportException[[gateway_59d729b0ac1b][10.50.15.26:9310]
handshake failed. unexpected remote node {data1}
{2wlGWuROTRGQ271YgCzQbQ}{t5UWdlbGTu67-Y42QSDJXA}
{10.50.15.26}{10.50.15.26:9310}{ml.machine_memory=16694579200,
ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]; ]
New it seems that data2 collides with data1 in some way.
Can somebody help me out?
It is – again – absolutely possible that there is a major lack in my understanding of this zen communication. Please explain then.
Thank you!