Zen Discovery and the transport ports

Elastic Stack Elasticsearch
1

Hi,

I maybe fail to understand the principles of zen discovery. But I can not make the Elasticsearch nodes join the cluster.

Info:

  • All Elasticsearch nodes live in Docker containers
  • Everything happens on my local machine for testing purpose

Now I want 2 master-only nodes to form a cluster. Here the relevant part of the config:

network.bind_host: 0.0.0.0
network.publish_host: 10.50.15.26      <– my local ip address
discovery.zen.minimum_master_nodes: 2

The first two lines are important for nodes inside Docker containers to communicate with each other and are less important for Zen Discovery I guess.

Without configuring transport.tcp.port and discovery.zen.ping.unicast.hosts the nodes fail to start.

This is how I start the containers:
docker run --name master1 -p 9300:9300 <elasticsearch_image>
docker run --name master2 -p 9301:9301 <elasticsearch_image>

This what I got:

[2019-01-29T14:40:15,370][WARN][o.e.d.z.ZenDiscovery] [master1]
not enough master nodes discovered during pinging
(found [[Candidate{node={master1}{fLG4b6OESXqlRPZMlU8Jyg}
{nBpqu7D9SzSme3-nD4JX3A}{10.50.15.26}{10.50.15.26:9300}
{ml.machine_memory=16694579200, xpack.installed=true, ml.max_open_jobs=20,
ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again

It seems as if master1 found itself and stopped looking for other nodes. Same for master2.

Now I do some configuration like this:
master1:

transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: 10.50.15.26:9301

master2:

transport.tcp.port: 9301
discovery.zen.ping.unicast.hosts: 10.50.15.26:9300

With this config everything runs fine.

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

While it can be reasonable to stick with this configuration for the master-only nodes (as they tend to stay in threes) it only shifts the problem.
Let's say I want to a bunch of data nodes or coordinating-only nodes. The configuration looks like this:

network.bind_host: 0.0.0.0
network.publish_host: 10.50.15.26
http.port: 9210-9219
transport.tcp.port: 9310-9319
discovery.zen:
  minimum_master_nodes: 2
  ping.unicast.hosts:
    - 10.50.15.26:9300      <– master1
    - 10.50.15.26:9301      <– master2
    - 10.50.15.26:9302      <– master3

Start them:
docker run --name data1 -p 9210:9210 -p 9310:9310 <elasticsearch_image>
docker run --name data2 -p 9211:9211 -p 9311:9311 <elasticsearch_image>
...
The first one successfully joins the cluster! The second gives this error:

[2019-01-29T14:46:29,157][INFO ][o.e.d.z.ZenDiscovery] [data2]
failed to send join request to master [{master2}{1bhY71VgS_6WIhg1Xu6ZIw}
{NyojrkngTDamazzfeDfHjg}{10.50.15.26}{10.50.15.26:9301}
{ml.machine_memory=16694579200, ml.max_open_jobs=20,
xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException
[[master2][172.24.0.4:9301][internal:discovery/zen/join]]; nested:
ConnectTransportException[[gateway_59d729b0ac1b][10.50.15.26:9310]
handshake failed. unexpected remote node {data1}
{2wlGWuROTRGQ271YgCzQbQ}{t5UWdlbGTu67-Y42QSDJXA}
{10.50.15.26}{10.50.15.26:9310}{ml.machine_memory=16694579200,
ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]; ]

New it seems that data2 collides with data1 in some way.
Can somebody help me out?
It is – again – absolutely possible that there is a major lack in my understanding of this zen communication. Please explain then.

Thank you!

2

Hi again,

it is probably unclear, what I want to achieve.
I try to say it in a few sentences:

  1. We want to scale dockerized Elasticsearch nodes horizontally.
  2. Potentially, multiple nodes can live on the same host.
  3. There has to be a configuration so that they automatically use a port between 93xx and 93yy without conflicting with other instances on the same host.
  4. They eventually need to discover the cluster.

Please help me finding this configuration.
Regards

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.