Unicast issue with docker elasticsearch container on two VMs

I have two ES docker container running on two VMs.

config: on VM 1

*path: logs: /data/log data: /data/data plugins: /data/plugins
work: /data/work cluster.name: t1-es-gld node: name: tg-es-pam-01
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["tg-es-pam-01", "tg-es-pam-02"]unicast
issue with docker elasticsearch container on two VMs. *

config: on VM2

path: logs: /data/log data: /data/data plugins: /data/plugins
work: /data/work cluster.name: t1-es-gld node: name: tg-es-pam-02
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["tg-es-pam-01", "tg-es-pam-02"]

Errors on VM 1

[2015-02-26 18:33:43,647][INFO ][cluster.service ] [tg-es-pam-01]
new_master
[tg-es-pam-01][tIpfaAAqQcGAzeA-OIY5JA][d396350dc668][inet[/172.17.0.2:9300]],
reason: zen-disco-join (elected_as_master) [2015-02-26 18:33:43,700][INFO
][gateway ] [tg-es-pam-01] recovered [0] indices into
cluster_state [2015-02-26 18:45:45,391][INFO ][node ]
[tg-es-pam-01] version[1.4.4], pid[1], build[c88f77f/2015-02-19T13:05:36Z]
[2015-02-26 18:45:45,392][INFO ][node ] [tg-es-pam-01]
initializing ... [2015-02-26 18:45:45,397][INFO ][plugins
] [tg-es-pam-01] loaded [], sites [] [2015-02-26 18:45:48,689][INFO ][node
] [tg-es-pam-01] initialized [2015-02-26
18:45:48,689][INFO ][node ] [tg-es-pam-01] starting ...
[2015-02-26 18:45:48,770][INFO ][transport ] [tg-es-pam-01]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/172.17.0.3:9300]} [2015-02-26 18:45:48,780][INFO ][discovery
] [tg-es-pam-01] t1-es-gld/Bl0pFprtToeWDUMqi-C6DA [2015-02-26
18:45:52,137][INFO ][discovery.zen ] [tg-es-pam-01] failed to
send join request to master
[[tg-es-pam-02][1kmhsjwfRsyyowmG42XnJg][d31804281a9c][inet[/172.17.0.3:9300]]],
reason
[RemoteTransportException[[tg-es-pam-01][inet[/172.17.0.3:9300]][internal:discovery/zen/join]];
nested: ElasticsearchIllegalStateException[Node
[[tg-es-pam-01][Bl0pFprtToeWDUMqi-C6DA][eafa8f60b3cb][inet[/172.17.0.3:9300]]]
not master for join request from
[[tg-es-pam-01][Bl0pFprtToeWDUMqi-C6DA][eafa8f60b3cb][inet[/172.17.0.3:9300]]]];
], tried [3] times

Errors on VM 2

*[2015-02-26 18:57:47,148][WARN ][discovery.zen ] [tg-es-pam-02]
failed to connect to master
[[tg-es-pam-01][Bl0pFprtToeWDUMqi-C6DA][eafa8f60b3cb][inet[/172.17.0.3:9300]]],
retrying... org.elasticsearch.transport.ConnectTransportException:
[tg-es-pam-01][inet[/172.17.0.3:9300]] connect_timeout[30s] at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:807)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:741)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:714)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:150)
at
org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:441)
at
org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:393<nabble_embed></nabble_embed>)
at
org.elasticsearch.discovery.zen.ZenDiscovery.access$6000(ZenDiscovery.java:80)
at
org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1320)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745) Caused by:
org.elasticsearch.common.netty.channel.ConnectTimeoutException: connection
timed out: /172.17.0.3:9300 at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:139)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
*

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f567939d-f47b-4839-98fa-c6f76c3a492f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Can you ping each node from the other? Telnet to 9200 and 9300?

If not then it's a docker issue and not an Elasticsearch one.

On 27 February 2015 at 06:58, Zicheng Wang wzcwts521@gmail.com wrote:

I have two ES docker container running on two VMs.

config: on VM 1

*path: logs: /data/log data: /data/data plugins: /data/plugins
work: /data/work cluster.name: t1-es-gld node: name: tg-es-pam-01
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["tg-es-pam-01", "tg-es-pam-02"]unicast
issue with docker elasticsearch container on two VMs. *

config: on VM2

path: logs: /data/log data: /data/data plugins: /data/plugins
work: /data/work cluster.name: t1-es-gld node: name: tg-es-pam-02
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["tg-es-pam-01", "tg-es-pam-02"]

Errors on VM 1

[2015-02-26 18:33:43,647][INFO ][cluster.service ]
[tg-es-pam-01] new_master
[tg-es-pam-01][tIpfaAAqQcGAzeA-OIY5JA][d396350dc668][inet[/172.17.0.2:9300]],
reason: zen-disco-join (elected_as_master) [2015-02-26 18:33:43,700][INFO
][gateway ] [tg-es-pam-01] recovered [0] indices into
cluster_state [2015-02-26 18:45:45,391][INFO ][node ]
[tg-es-pam-01] version[1.4.4], pid[1], build[c88f77f/2015-02-19T13:05:36Z]
[2015-02-26 18:45:45,392][INFO ][node ] [tg-es-pam-01]
initializing ... [2015-02-26 18:45:45,397][INFO ][plugins
] [tg-es-pam-01] loaded [], sites [] [2015-02-26 18:45:48,689][INFO ][node
] [tg-es-pam-01] initialized [2015-02-26
18:45:48,689][INFO ][node ] [tg-es-pam-01] starting ...
[2015-02-26 18:45:48,770][INFO ][transport ] [tg-es-pam-01]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/172.17.0.3:9300 http://172.17.0.3:9300]} [2015-02-26
18:45:48,780][INFO ][discovery ] [tg-es-pam-01]
t1-es-gld/Bl0pFprtToeWDUMqi-C6DA [2015-02-26 18:45:52,137][INFO
][discovery.zen ] [tg-es-pam-01] failed to send join request to
master
[[tg-es-pam-02][1kmhsjwfRsyyowmG42XnJg][d31804281a9c][inet[/172.17.0.3:9300]]],
reason
[RemoteTransportException[[tg-es-pam-01][inet[/172.17.0.3:9300]][internal:discovery/zen/join]];
nested: ElasticsearchIllegalStateException[Node
[[tg-es-pam-01][Bl0pFprtToeWDUMqi-C6DA][eafa8f60b3cb][inet[/172.17.0.3:9300]]]
not master for join request from
[[tg-es-pam-01][Bl0pFprtToeWDUMqi-C6DA][eafa8f60b3cb][inet[/172.17.0.3:9300]]]];
], tried [3] times

Errors on VM 2

*[2015-02-26 18:57:47,148][WARN ][discovery.zen ]
[tg-es-pam-02] failed to connect to master
[[tg-es-pam-01][Bl0pFprtToeWDUMqi-C6DA][eafa8f60b3cb][inet[/172.17.0.3:9300]]],
retrying... org.elasticsearch.transport.ConnectTransportException:
[tg-es-pam-01][inet[/172.17.0.3:9300 http://172.17.0.3:9300]]
connect_timeout[30s] at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:807)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:741)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:714)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:150)
at
org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:441)
at
org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:393<nabble_embed></nabble_embed>)
at
org.elasticsearch.discovery.zen.ZenDiscovery.access$6000(ZenDiscovery.java:80)
at
org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1320)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745) Caused by:
org.elasticsearch.common.netty.channel.ConnectTimeoutException: connection
timed out: /172.17.0.3:9300 at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:139)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
*

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f567939d-f47b-4839-98fa-c6f76c3a492f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f567939d-f47b-4839-98fa-c6f76c3a492f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-_W3_HiAixeYpSfGASScSv7oXX%3DDnR%3DjNb2CZczXy%2Bwg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Thanks Mark

I'm able to telnet each node from the other for both port. :frowning:

from VM 2 to VM 1
[root@tg-es-pam-02 data]# telnet tg-es-pam-01 9200
Trying 192.168.xx.xx...
Connected to tg-es-pam-01.xxxxx (192.168.xx.xx).

[root@tg-es-pam-02 data]# telnet tg-es-pam-01 9300
Trying 192.168.xx.xx...
Connected to tg-es-pam-01.xxxxxx (192.168.xx.xx).

from VM 1 to VM 2
[root@tg-es-pam-01 log]# telnet tg-es-pam-02 9300
Trying 192.168.xx.xx...
Connected to tg-es-pam-02.xxxxxx (192.168.xx.xx).
Escape character is '^]'.

Connection closed by foreign host.
[root@tg-es-pam-01 log]# telnet tg-es-pam-02 9200
Trying 192.168.xx.xx...
Connected to tg-es-pam-02.xxxxxx (192.168.xx.xx).

On Thursday, February 26, 2015 at 12:55:23 PM UTC-8, Mark Walkom wrote:

Can you ping each node from the other? Telnet to 9200 and 9300?

If not then it's a docker issue and not an Elasticsearch one.

On 27 February 2015 at 06:58, Zicheng Wang <wzcw...@gmail.com
<javascript:>> wrote:

I have two ES docker container running on two VMs.

config: on VM 1

*path: logs: /data/log data: /data/data plugins: /data/plugins
work: /data/work cluster.name: t1-es-gld node: name: tg-es-pam-01
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["tg-es-pam-01", "tg-es-pam-02"]unicast
issue with docker elasticsearch container on two VMs. *

config: on VM2

path: logs: /data/log data: /data/data plugins: /data/plugins
work: /data/work cluster.name: t1-es-gld node: name: tg-es-pam-02
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["tg-es-pam-01", "tg-es-pam-02"]

Errors on VM 1

[2015-02-26 18:33:43,647][INFO ][cluster.service ]
[tg-es-pam-01] new_master
[tg-es-pam-01][tIpfaAAqQcGAzeA-OIY5JA][d396350dc668][inet[/172.17.0.2:9300]],
reason: zen-disco-join (elected_as_master) [2015-02-26 18:33:43,700][INFO
][gateway ] [tg-es-pam-01] recovered [0] indices into
cluster_state [2015-02-26 18:45:45,391][INFO ][node ]
[tg-es-pam-01] version[1.4.4], pid[1], build[c88f77f/2015-02-19T13:05:36Z]
[2015-02-26 18:45:45,392][INFO ][node ] [tg-es-pam-01]
initializing ... [2015-02-26 18:45:45,397][INFO ][plugins
] [tg-es-pam-01] loaded [], sites [] [2015-02-26 18:45:48,689][INFO ][node
] [tg-es-pam-01] initialized [2015-02-26
18:45:48,689][INFO ][node ] [tg-es-pam-01] starting ...
[2015-02-26 18:45:48,770][INFO ][transport ] [tg-es-pam-01]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/172.17.0.3:9300 http://172.17.0.3:9300]} [2015-02-26
18:45:48,780][INFO ][discovery ] [tg-es-pam-01]
t1-es-gld/Bl0pFprtToeWDUMqi-C6DA [2015-02-26 18:45:52,137][INFO
][discovery.zen ] [tg-es-pam-01] failed to send join request to
master
[[tg-es-pam-02][1kmhsjwfRsyyowmG42XnJg][d31804281a9c][inet[/172.17.0.3:9300]]],
reason
[RemoteTransportException[[tg-es-pam-01][inet[/172.17.0.3:9300]][internal:discovery/zen/join]];
nested: ElasticsearchIllegalStateException[Node
[[tg-es-pam-01][Bl0pFprtToeWDUMqi-C6DA][eafa8f60b3cb][inet[/172.17.0.3:9300]]]
not master for join request from
[[tg-es-pam-01][Bl0pFprtToeWDUMqi-C6DA][eafa8f60b3cb][inet[/172.17.0.3:9300]]]];
], tried [3] times

Errors on VM 2

*[2015-02-26 18:57:47,148][WARN ][discovery.zen ]
[tg-es-pam-02] failed to connect to master
[[tg-es-pam-01][Bl0pFprtToeWDUMqi-C6DA][eafa8f60b3cb][inet[/172.17.0.3:9300]]],
retrying... org.elasticsearch.transport.ConnectTransportException:
[tg-es-pam-01][inet[/172.17.0.3:9300 http://172.17.0.3:9300]]
connect_timeout[30s] at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:807)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:741)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:714)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:150)
at
org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:441)
at
org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:393<nabble_embed></nabble_embed>)
at
org.elasticsearch.discovery.zen.ZenDiscovery.access$6000(ZenDiscovery.java:80)
at
org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1320)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745) Caused by:
org.elasticsearch.common.netty.channel.ConnectTimeoutException: connection
timed out: /172.17.0.3:9300 at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:139)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
*

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f567939d-f47b-4839-98fa-c6f76c3a492f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f567939d-f47b-4839-98fa-c6f76c3a492f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1a7968c1-8101-40d8-b50f-52b7ed6e2cef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.