Hi,
I'm trying to setup a 2 node cluster. one of the elastic node is running inside docker VM. My docker vm itself is running inside Virtual box vm. And the other node works inside my host machine - windows.
If i run each of the nodes in standalone (disable cluster), i'm able to ping the elastic search UI from the other machine. But when both these node are started to join the same cluster, zen discovery mechanism fails with the an error.
I have disabled firewall extra without any luck. see below the configuration used in both my node. I can able to ping each of the nodes from the other node. Docker VM was started with port forwarding for 9200 and 9300
Node - 1
cluster.name: my-cluster
network.host: --Node -1 IP --
discovery.zen.ping.unicast.hosts: ["--Node -2 IP :9300"]
discovery.zen.ping.unicast.hosts.resolve_timeout: 100s
discovery.zen.ping_timeout: 10s
discovery.zen.minimum_master_nodes: 2
Node - 2
cluster.name: my-cluster
network.host: --Node -2 IP --
discovery.zen.ping.unicast.hosts: ["--Node -1 IP :9300"]
discovery.zen.ping.unicast.hosts.resolve_timeout: 100s
discovery.zen.ping_timeout: 10s
discovery.zen.minimum_master_nodes: 2
Log from node running in Host machine -- Windows.
[2018-12-14T14:27:36,222][TRACE][o.e.d.z.UnicastZenPing ] [pcE7vKt] [1] failed to ping {192.168.33.10:9300}{XX3lS5kJR2yQ_T-AWeXbcQ}{192.168.33.10}{192.168.33.10:9300}
org.elasticsearch.transport.ConnectTransportException: [][192.168.33.10:9300] connect_exception
at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) ~[elasticsearch-6.4.2.jar:6.4.2]
..............................
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: /[Node 2 IP]:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
... 1 more
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) ~[?:?]
Also somewhere down the log i also see this..
[2018-12-14T14:27:51,884][TRACE][o.e.t.n.ESLoggingHandler ] an exception occurred formatting a trace message
java.io.EOFException: tried to read: 105 bytes but only 22 remaining
at org.elasticsearch.transport.netty4.ByteBufStreamInput.ensureCanReadBytes(ByteBufStreamInput.java:75) ~[?:?]
at org.elasticsearch.common.io.stream.StreamInput.readArraySize(StreamInput.java:1026) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.common.io.stream.StreamInput.readString(StreamInput.java:374) ~[elasticsearch-6.4.2.jar:6.4.2]
..................
[2018-12-14T14:27:51,886][TRACE][o.e.t.n.ESLoggingHandler ] [id: 0x62c3a99c, L:/192.168.33.1:59012 - R:/192.168.33.10:9300] WRITE: 45B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 45 53 00 00 00 27 00 00 00 00 00 00 00 07 08 00 |ES...'..........|
|00000010| 4d 36 03 00 00 16 69 6e 74 65 72 6e 61 6c 3a 74 |M6....internal:t|
|00000020| 63 70 2f 68 61 6e 64 73 68 61 6b 65 00 |cp/handshake. |
+--------+-------------------------------------------------+----------------+
[2018-12-14T14:27:51,895][TRACE][o.e.x.s.t.n.SecurityNetty4ServerTransport] [pcE7vKt] Tcp transport channel accepted: NettyTcpChannel{localAddress=/192.168.33.1:9300, remoteAddress=/[node 2 IP:56046}