Zen Discovery and the transport ports


I maybe fail to understand the principles of zen discovery. But I can not make the Elasticsearch nodes join the cluster.


  • All Elasticsearch nodes live in Docker containers
  • Everything happens on my local machine for testing purpose

Now I want 2 master-only nodes to form a cluster. Here the relevant part of the config:

network.publish_host:      <– my local ip address
discovery.zen.minimum_master_nodes: 2

The first two lines are important for nodes inside Docker containers to communicate with each other and are less important for Zen Discovery I guess.

Without configuring transport.tcp.port and discovery.zen.ping.unicast.hosts the nodes fail to start.

This is how I start the containers:
docker run --name master1 -p 9300:9300 <elasticsearch_image>
docker run --name master2 -p 9301:9301 <elasticsearch_image>

This what I got:

[2019-01-29T14:40:15,370][WARN][o.e.d.z.ZenDiscovery] [master1]
not enough master nodes discovered during pinging
(found [[Candidate{node={master1}{fLG4b6OESXqlRPZMlU8Jyg}
{ml.machine_memory=16694579200, xpack.installed=true, ml.max_open_jobs=20,
ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again

It seems as if master1 found itself and stopped looking for other nodes. Same for master2.

Now I do some configuration like this:

transport.tcp.port: 9300


transport.tcp.port: 9301

With this config everything runs fine.


While it can be reasonable to stick with this configuration for the master-only nodes (as they tend to stay in threes) it only shifts the problem.
Let's say I want to a bunch of data nodes or coordinating-only nodes. The configuration looks like this:

http.port: 9210-9219
transport.tcp.port: 9310-9319
  minimum_master_nodes: 2
    -      <– master1
    -      <– master2
    -      <– master3

Start them:
docker run --name data1 -p 9210:9210 -p 9310:9310 <elasticsearch_image>
docker run --name data2 -p 9211:9211 -p 9311:9311 <elasticsearch_image>
The first one successfully joins the cluster! The second gives this error:

[2019-01-29T14:46:29,157][INFO ][o.e.d.z.ZenDiscovery] [data2]
failed to send join request to master [{master2}{1bhY71VgS_6WIhg1Xu6ZIw}
{ml.machine_memory=16694579200, ml.max_open_jobs=20,
xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException
[[master2][][internal:discovery/zen/join]]; nested:
handshake failed. unexpected remote node {data1}
ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]; ]

New it seems that data2 collides with data1 in some way.
Can somebody help me out?
It is – again – absolutely possible that there is a major lack in my understanding of this zen communication. Please explain then.

Thank you!

Hi again,

it is probably unclear, what I want to achieve.
I try to say it in a few sentences:

  1. We want to scale dockerized Elasticsearch nodes horizontally.
  2. Potentially, multiple nodes can live on the same host.
  3. There has to be a configuration so that they automatically use a port between 93xx and 93yy without conflicting with other instances on the same host.
  4. They eventually need to discover the cluster.

Please help me finding this configuration.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.