Failure to elect master, v7.17

I'm pulling my hair out trying to figure out why my three nodes, running as docker-compose containers on 3 separate Ubuntu 22.04LTS hosts, cannot start.

The 3 nodes discover each other but fail to elect a master. When I curl to get the list of nodes, I get a "master not discovered exception".

Here are the repeating log statements on the 3 nodes:

node01-master    | {"type": "server", "timestamp": "2023-08-04T15:54:14,467Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "es-cluster-test-73", "node.name": "node01-master", "message": "master not discovered or elected yet, an election requires at least 2 nodes with ids from [XSnNy2f2QOmu9ZrAbEqeGA, TWCTna4GRzSk4NQmYLe8aw, CrOUreENRAKBvUHmMSqIlA], have only discovered non-quorum [{node01-master}{CrOUreENRAKBvUHmMSqIlA}{ODyLuSnYSNusYwdWUa-tFg}{172.16.0.152}{172.16.0.152:9307}{cdfhimrstw}, {node02-data}{PjWpoi54RIuHbTbNN2eqzQ}{NnQM-vFpQd6nC-BQa5QFGg}{172.16.0.149}{172.16.0.149:9307}{cdfhimrstw}, {node03-data}{gcWZXovdQGST_aVd1sedhQ}{YYN3f3gbTcS0_mDrfoasGg}{172.16.0.148}{172.16.0.148:9307}{cdfhimrstw}]; discovery will continue using [172.16.0.149:9307, 172.16.0.148:9307] from hosts providers and [{node01-master}{CrOUreENRAKBvUHmMSqIlA}{ODyLuSnYSNusYwdWUa-tFg}{172.16.0.152}{172.16.0.152:9307}{cdfhimrstw}] from last-known cluster state; node term 2, last-accepted version 101 in term 2" }

As you can see, one of the two expected IDs has been discovered.

node02-data    | {"type": "server", "timestamp": "2023-08-04T15:54:25,534Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "es-cluster-test-73", "node.name": "node02-data", "message": "master not discovered or elected yet, an election requires two nodes with ids [PjWpoi54RIuHbTbNN2eqzQ, CrOUreENRAKBvUHmMSqIlA], have discovered possible quorum [{node02-data}{PjWpoi54RIuHbTbNN2eqzQ}{NnQM-vFpQd6nC-BQa5QFGg}{172.16.0.149}{172.16.0.149:9307}{cdfhimrstw}, {node01-master}{CrOUreENRAKBvUHmMSqIlA}{ODyLuSnYSNusYwdWUa-tFg}{172.16.0.152}{172.16.0.152:9307}{cdfhimrstw}, {node03-data}{gcWZXovdQGST_aVd1sedhQ}{YYN3f3gbTcS0_mDrfoasGg}{172.16.0.148}{172.16.0.148:9307}{cdfhimrstw}]; discovery will continue using [172.16.0.152:9307, 172.16.0.148:9307] from hosts providers and [{node02-data}{PjWpoi54RIuHbTbNN2eqzQ}{NnQM-vFpQd6nC-BQa5QFGg}{172.16.0.149}{172.16.0.149:9307}{cdfhimrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }

As you can see, both of the two expected IDs have been discovered.

node03-data    | {"type": "server", "timestamp": "2023-08-04T15:54:35,075Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "es-cluster-test-73", "node.name": "node03-data", "message": "master not discovered or elected yet, an election requires 2 nodes with ids [gcWZXovdQGST_aVd1sedhQ, CrOUreENRAKBvUHmMSqIlA], have discovered possible quorum [{node03-data}{gcWZXovdQGST_aVd1sedhQ}{YYN3f3gbTcS0_mDrfoasGg}{172.16.0.148}{172.16.0.148:9307}{cdfhimrstw}, {node01-master}{CrOUreENRAKBvUHmMSqIlA}{ODyLuSnYSNusYwdWUa-tFg}{172.16.0.152}{172.16.0.152:9307}{cdfhimrstw}, {node02-data}{PjWpoi54RIuHbTbNN2eqzQ}{NnQM-vFpQd6nC-BQa5QFGg}{172.16.0.149}{172.16.0.149:9307}{cdfhimrstw}]; discovery will continue using [172.16.0.152:9307, 172.16.0.149:9307] from hosts providers and [{node03-data}{gcWZXovdQGST_aVd1sedhQ}{YYN3f3gbTcS0_mDrfoasGg}{172.16.0.148}{172.16.0.148:9307}{cdfhimrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }

As you can see, again, both of the two expected IDs have been discovered.

I don't know how important the last line is in each log line, about the "last-accepted version" cluster state and the "term". The 1st node keeps on giving the same values for those. The other two are at 0 for both.

Here is the docker-compose.yaml for the 1st node (the others are of course very similar). The IP addresses you see belong to the docker hosts.

version: '3.7'
services:
  node01-master:
    build: .
    container_name: node01-master
    hostname: es-master
    environment:
      - node.name=node01-master
      - cluster.name=es-cluster-test-73
      - discovery.seed_hosts=172.16.0.152:9307,172.16.0.149:9307,172.16.0.148:9307
      - cluster.initial_master_nodes=node01-master,node02-data,node03-data
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms256M -Xmx256M"
      - node.master=true
      - node.voting_only=false
      - node.data=true
      - node.ingest=true
      - node.ml=false
      - xpack.ml.enabled=true
      - cluster.remote.connect=true
      - network.publish_host=172.16.0.152
      - transport.publish_port=9307
      - http.publish_port=9307
    volumes:
      - data01:/usr/share/elasticsearch/data
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - "9207:9200"
      - "9307:9300"
    networks:
      - elastic_73
    restart: always

volumes:
  data01:
    driver: local

networks:
  elastic_73:
    driver: bridge
    ipam:
      driver: default
      config:
      - subnet: 10.40.4.1/24

I'd be grateful for any help!

  • George

Can you provide a little more context about your infrastructure? It is a little confusing.

You have three hosts, and in each one of them you are using a docker-compose to spin up an elasticsearch node, right?

In your discovery.seed_hosts you have this:

172.16.0.152:9307,172.16.0.149:9307,172.16.0.148:9307

Are those IP addresses the IP for the Ubuntu hosts?

This looks like a docker network issue, not an Elasticsearch one.

Any reason to use this? I do not know much about docker, but is this IP address available inside the container? I would remove this and just use network.host: 0.0.0.0, this would bind Elasticsearch to the private IP address ofthe container.

Also, not sure what is happening here, the http and transport port should be different, but I don't think this config is needed at all, you are already exposing the port 9300, so any request to your hosts on port 9307 will be redirect to the port 9300 in the container.

I would remove this:

      - transport.publish_port=9307
      - http.publish_port=9307

And change this:

      - network.publish_host=172.16.0.152

To this:

    - network.host: 0.0.0.0

Thank you very much for responding!

Yes, as mentioned the IPs are the host IPs.

I made the changes you suggested and now we have a different problem. It seems to not like the docker container IP addresses.

node03-data | {"type": "server", "timestamp": "2023-08-04T18:02:14,399Z", "level": "WARN", "component": "o.e.d.HandshakingTransportAddressConnector", "cluster.name": "es-cluster-test-733", "node.name": "node03-data", "message": "[connectToRemoteMasterNode[172.16.0.152:9307]] completed handshake with [{node01-master}{CrOUreENRAKBvUHmMSqIlA}{lRPcwhDvQmOyaMLoHPOmTw}{10.40.4.2}{10.40.4.2:9300}{cdfhilmrstw}{ml.machine_memory=4102238208, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=268435456, transform.node=true}] but followup connection failed",

node03-data | "stacktrace": ["org.elasticsearch.transport.ConnectTransportException: [node01-master][10.40.4.2:9300] handshake failed. unexpected remote node {node03-data}{gcWZXovdQGST_aVd1sedhQ}{NrCtzu0CSQ26D49iCKh06A}{10.40.4.2}{10.40.4.2:9300}{cdfhilmrstw}{ml.machine_memory=4102238208, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=268435456, transform.node=true}",

The ammended YAML is:

version: '3.7'
services:
  node01-master:
    build: .
    container_name: node01-master
    hostname: es-master
    environment:
      - node.name=node01-master
      - cluster.name=es-cluster-test-733
      - discovery.seed_hosts=172.16.0.152:9307,172.16.0.149:9307,172.16.0.148:9307
      - cluster.initial_master_nodes=node01-master,node02-data,node03-data
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms256M -Xmx256M"
      - node.master=true
      # - node.voting_only=false
      - node.data=true
      - node.ingest=true
      # - node.ml=false
      - xpack.ml.enabled=true
      - cluster.remote.connect=true
      - network.host=0.0.0.0

    volumes:
      - data01:/usr/share/elasticsearch/data
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - "9207:9200"
      - "9307:9300"
    networks:
      - elastic_73
    restart: always

volumes:
  data01:
    driver: local

networks:
  elastic_73:
    driver: bridge
    ipam:
      driver: default
      config:
      - subnet: 10.40.4.1/24

Now it seems that the nodes are able to talk with each other, but are returning the container IP.

It seems that you the network.publish_host is need indeed, but also the network.host.

Try to use this in the yaml files:

  - network.host=0.0.0.0
  - network.publish_host=host-ip-address

It seems to have worked on this simular post.

OK, with your suggestion, the IP is the host IP but the port is the container port, so again we have a problem:

node03-data | {"type": "server", "timestamp": "2023-08-04T19:39:00,563Z", "level": "WARN", "component": "o.e.d.HandshakingTransportAddressConnector", "cluster.name": "es-cluster-test-733", "node.name": "node03-data", "message": "[connectToRemoteMasterNode[172.16.0.149:9307]] completed handshake with [{node02-data}{PjWpoi54RIuHbTbNN2eqzQ}{AcNpq_EVRhuLLHGfon7NeQ}{172.16.0.149}{172.16.0.149:9300}{cdfhilmrstw}{ml.machine_memory=4102238208, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=268435456, transform.node=true}] but followup connection failed",

Try to set the cluster.initial_master_nodes the same as the discover.seed_hosts.

      - discovery.seed_hosts=172.16.0.152:9307,172.16.0.149:9307,172.16.0.148:9307
      - cluster.initial_master_nodes=172.16.0.152:9307,172.16.0.149:9307,172.16.0.148:9307

This cluster has already formed once so you must not set cluster.initial_master_nodes.

What is the rest of the log message?

It is still using 9300 and the host IP so no-go...

node01-master | {"type": "server", "timestamp": "2023-08-04T21:17:01,492Z", "level": "WARN", "component": "o.e.d.HandshakingTransportAddressConnector", "cluster.name": "es-cluster-test-733", "node.name": "node01-master", "message": "[connectToRemoteMasterNode[172.16.0.149:9307]] completed handshake with [{node02-data}{PjWpoi54RIuHbTbNN2eqzQ}{NzmWfxD3Q1uwztuR5aDk3g}{172.16.0.149}{172.16.0.149:9300}{cdfhilmrstw}{ml.machine_memory=4102238208, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=268435456, transform.node=true}] but followup connection failed",

I think I do need to set transport.port=9307

Thanks for helping me out!

How can we consider the cluster started when it never elected a master?

In my very fist post I stressed that two of the 3 nodes had indeed discovered the IDs they were looking for, but still didn't elect a master. The one node that had not, was the 1st node, the one that had different "last-accepted version" 101 and term=2, when both others had 0 and 0. Do these numbers matter???

Not in your case, no, it's the rest of the followup connection failed message that matters, and you haven't shared the full message yet.

I expect it'll help to be consistent about the port you use everywhere, but why not use the default of 9300? That'd be simpler I reckon.

The port is 9307 because 9300 is taken.

Here is more logging:

node01-master    | {"type": "server", "timestamp": "2023-08-05T09:55:51,895Z", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "es-cluster-test-733", "node.name": "node01-master", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" }
node01-master    | {"type": "server", "timestamp": "2023-08-05T09:55:55,959Z", "level": "WARN", "component": "o.e.d.HandshakingTransportAddressConnector", "cluster.name": "es-cluster-test-733", "node.name": "node01-master", "message": "[connectToRemoteMasterNode[172.16.0.149:9307]] completed handshake with [{node02-data}{oZ6q6HffSNCd46UNKbuZxQ}{XoQRq75wRoaLPSuC8OIizA}{172.16.0.149}{172.16.0.149:9300}{cdfhilmrstw}{ml.machine_memory=4102238208, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=268435456, transform.node=true}] but followup connection failed",
node01-master    | "stacktrace": ["org.elasticsearch.transport.ConnectTransportException: [node02-data][172.16.0.149:9300] general node connection failure",
node01-master    | "at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.lambda$onResponse$2(TcpTransport.java:1035) ~[elasticsearch-7.17.7.jar:7.17.7]",
node01-master    | "at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:144) ~[elasticsearch-7.17.7.jar:7.17.7]",
node01-master    | "at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleLocalException(TransportHandshaker.java:155) ~[elasticsearch-7.17.7.jar:7.17.7]",
node01-master    | "at org.elasticsearch.transport.TransportHandshaker.lambda$sendHandshake$0(TransportHandshaker.java:52) ~[elasticsearch-7.17.7.jar:7.17.7]",
node01-master    | "at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:241) ~[elasticsearch-7.17.7.jar:7.17.7]",
node01-master    | "at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277) ~[elasticsearch-7.17.7.jar:7.17.7]",
node01-master    | "at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:28) ~[elasticsearch-core-7.17.7.jar:7.17.7]",
node01-master    | "at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]",
node01-master    | "at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]",
node01-master    | "at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]",
node01-master    | "at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?]",
node01-master    | "at org.elasticsearch.core.CompletableContext.complete(CompletableContext.java:50) ~[elasticsearch-core-7.17.7.jar:7.17.7]",
node01-master    | "at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:51) ~[?:?]",
node01-master    | "at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[?:?]",
node01-master    | "at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) ~[?:?]",
node01-master    | "at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) ~[?:?]",
node01-master    | "at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[?:?]",
node01-master    | "at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[?:?]",
node01-master    | "at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605) ~[?:?]",
node01-master    | "at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[?:?]",
node01-master    | "at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[?:?]",
node01-master    | "at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1182) ~[?:?]",
node01-master    | "at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:773) ~[?:?]",
node01-master    | "at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:749) ~[?:?]",
node01-master    | "at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:620) ~[?:?]",
node01-master    | "at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:105) ~[?:?]",
node01-master    | "at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:174) ~[?:?]",
node01-master    | "at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[?:?]",
node01-master    | "at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) ~[?:?]",
node01-master    | "at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) ~[?:?]",
node01-master    | "at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]",
node01-master    | "at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[?:?]",
node01-master    | "at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]",
node01-master    | "at java.lang.Thread.run(Thread.java:1589) [?:?]",
node01-master    | "Caused by: org.elasticsearch.transport.TransportException: handshake failed because connection reset",
node01-master    | "... 31 more"] }
node01-master    | {"type": "server", "timestamp": "2023-08-05T09:55:56,948Z", "level": "WARN", "component": "o.e.d.HandshakingTransportAddressConnector", "cluster.name": "es-cluster-test-733", "node.name": "node01-master", "message": "[connectToRemoteMasterNode[172.16.0.149:9307]] completed handshake with [{node02-data}{oZ6q6HffSNCd46UNKbuZxQ}{XoQRq75wRoaLPSuC8OIizA}{172.16.0.149}{172.16.0.149:9300}{cdfhilmrstw}{ml.machine_memory=4102238208, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=268435456, transform.node=true}] but followup connection failed",
node01-master    | "stacktrace": ["org.elasticsearch.transport.ConnectTransportException: [node02-data][172.16.0.149:9300] general node connection failure",
node01-master    | "at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.lambda$onResponse$2(TcpTransport.java:1035) ~[elasticsearch-7.17.7.jar:7.17.7]",
node01-master    | "at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:144) ~[elasticsearch-7.17.7.jar:7.17.7]",
node01-master    | "at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleLocalException(TransportHandshaker.java:155) ~[elasticsearch-7.17.7.jar:7.17.7]",
node01-master    | "at org.elasticsearch.transport.TransportHandshaker.lambda$sendHandshake$0(TransportHandshaker.java:52) ~[elasticsearch-7.17.7.jar:7.17.7]",
node01-master    | "at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:241) ~[elasticsearch-7.17.7.jar:7.17.7]",
node01-master    | "at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277) ~[elasticsearch-7.17.7.jar:7.17.7]",
node01-master    | "at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:28) ~[elasticsearch-core-7.17.7.jar:7.17.7]",
node01-master    | "at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]",
node01-master    | "at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]",
node01-master    | "at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]",
node01-master    | "at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?]",
node01-master    | "at org.elasticsearch.core.CompletableContext.complete(CompletableContext.java:50) ~[elasticsearch-core-7.17.7.jar:7.17.7]",
node01-master    | "at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:51) ~[?:?]",
node01-master    | "at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[?:?]",
node01-master    | "at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) ~[?:?]",
node01-master    | "at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) ~[?:?]",
node01-master    | "at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[?:?]",
node01-master    | "at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[?:?]",
node01-master    | "at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605) ~[?:?]",
node01-master    | "at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[?:?]",
node01-master    | "at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[?:?]",
node01-master    | "at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1182) ~[?:?]",
node01-master    | "at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:773) ~[?:?]",
node01-master    | "at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:749) ~[?:?]",
node01-master    | "at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:620) ~[?:?]",
node01-master    | "at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:105) ~[?:?]",
node01-master    | "at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:174) ~[?:?]",
node01-master    | "at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[?:?]",
node01-master    | "at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) ~[?:?]",
node01-master    | "at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) ~[?:?]",
node01-master    | "at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]",
node01-master    | "at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[?:?]",
node01-master    | "at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]",
node01-master    | "at java.lang.Thread.run(Thread.java:1589) [?:?]",
node01-master    | "Caused by: org.elasticsearch.transport.TransportException: handshake failed because connection reset",
node01-master    | "... 31 more"] }

This is the issue: you can connect to this node at 172.16.0.149:9307 but when it tries its publish address of 172.16.0.149:9300 it gets a Connection reset error. See these docs for more information, particularly:

[... each node must be] accessible at its transport publish address by all other nodes in its cluster ...

This cannot happen with docker. The ES node is exposed to outside traffic via port 9307. Port 9300 is an internal port. I would be used for intern-node communication if all nodes resided on the same host and communicated over the docket network. Since they do not, all traffic must take place over the exposed ports, in my case 9307, and the external host IPs.

In my 1st post I have shown how I got all nodes to discover each other. Two of them even found exactly the node IDs they were looking for. Still, no master was elected.

I wish we could continue from that point, where things were ALMOST working. All attempts since then have taken us backwards.

I quite agree. Can you go back to that state and start the troubleshooting again, only this time following the troubleshooting guide in the manual which points you to the specific log messages you need to study? I believe you were in the state described by this paragraph at the beginning:

If the logs or the health report indicate that Elasticsearch has discovered a possible quorum of nodes, the typical reason that the cluster can’t elect a master is that one of the other nodes can’t discover a quorum. Inspect the logs on the other master-eligible nodes and ensure that they have all discovered enough nodes to form a quorum.

But you did not share the relevant logs, and then the guesswork started.

The logs that you asked for yesterday (and got today), applied to a setup modified twice with two mods proposed by Leandro. And as you have seen that extra logging didn't help in any way.

I restored my initial config, deleted dangling docker volumes, started the cluster and it worked fine. It was correct right from the start. It was failing because of left-over config that was inside the volumes. With a clean restart it all worked.

Also, restarting the cluster without removing the "initial cluster nodes" setting did no harm at all. The indices I had already created were right there after the restart.

Thanks to both who tried to help me out. My (working) config is at the top of this thread, for those who will need it in the future.

Elastic ought to be providing such a config in their docs but apparently they don't care enough. They only provide a config for a single-host 3-node cluster, i.e. a useless setup.

That's not the case. The logs you shared pinpoint the exact problem with the setup to which they related, which I turned into words in my earlier post. If you'd shared the full logs from the original config then we would have been able to give an equally precise diagnosis too.

That's normally the case, but it's still best to remove it. These docs describe the risk you are taking by leaving it in place:

If you leave cluster.initial_master_nodes in place once the cluster has formed then there is a risk that a future misconfiguration may result in bootstrapping a new cluster alongside your existing cluster. It may not be possible to recover from this situation without losing data.

That is the case. I provided logs from my initial setup, and you did not ask for extra logs from that setup. You asked for extra logs after I had modified the setup twice and had provided logs for those two modified setups, therefore asking for "the rest of the log message" could only refer to the latest logging message, not the initial one.

You keep on obsessing with these logs, even after the thread got resolved. That's just bizarre.

Goodbye!

Yeah sorry this is more for future readers and @leandrojmp than for @bobus: please don't just guess at random config changes without gathering the appropriate information first. It's frustrating as a user to be given suggestions that don't help, and which may make things worse. The information you need to work out the actual problem, and its solution, is there in the logs.

@DavidTurner, I disagree on that, the issue, as mentioned by the OP was some left-over data inside the docker volumes and the solution was to remove all this data and start the cluster again, this information was never provided and all the logs suggested network issues.

If you can point out the exactly log line that was shared and would mean that the issue is left-over data from a previous cluster formation it would help in future troubleshoots.

I don't think clearing out all the old data really was the right solution, although clearly was a solution in this case. I must admit I don't really understand how it fixed the symptoms, I think there was more to this situation than we saw. Generally we wouldn't expect users to clear out old data in this kind of situation, at least we hadn't seen anything to indicate that it was unrecoverable.

Everything I saw was also consistent with it being a network issue, but the exact nature of the network issue is important and we didn't see that until everything had changed. Your first post very quickly suggested network.host: 0.0.0.0 and removing the transport.publish_port setting, but I believe these were not appropriate suggestions given the information available at that time, and the rest of the thread showed that they were in fact not the right things to do. You later suggested setting cluster.initial_master_nodes, but this was certainly not correct given the logs shared earlier.