I have two ES docker container running on two VMs.
config: on VM 1
path:
logs: /data/log
data: /data/data
plugins: /data/plugins
work: /data/work
cluster.name: t1-es-gld
node:
name: tg-es-pam-01
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["tg-es-pam-01", "tg-es-pam-02"]
config: on VM2
path:
logs: /data/log
data: /data/data
plugins: /data/plugins
work: /data/work
cluster.name: t1-es-gld
node:
name: tg-es-pam-02
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["tg-es-pam-01", "tg-es-pam-02"]
Errors on VM 1
[2015-02-26 18:33:43,647][INFO ][cluster.service ] [tg-es-pam-01] new_master [tg-es-pam-01][tIpfaAAqQcGAzeA-OIY5JA][d396350dc668][inet[/172.17.0.2:9300]], reason: zen-disco-join (elected_as_master)
[2015-02-26 18:33:43,700][INFO ][gateway ] [tg-es-pam-01] recovered [0] indices into cluster_state
[2015-02-26 18:45:45,391][INFO ][node ] [tg-es-pam-01] version[1.4.4], pid[1], build[c88f77f/2015-02-19T13:05:36Z]
[2015-02-26 18:45:45,392][INFO ][node ] [tg-es-pam-01] initializing ...
[2015-02-26 18:45:45,397][INFO ][plugins ] [tg-es-pam-01] loaded [], sites []
[2015-02-26 18:45:48,689][INFO ][node ] [tg-es-pam-01] initialized
[2015-02-26 18:45:48,689][INFO ][node ] [tg-es-pam-01] starting ...
[2015-02-26 18:45:48,770][INFO ][transport ] [tg-es-pam-01] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/172.17.0.3:9300]}
[2015-02-26 18:45:48,780][INFO ][discovery ] [tg-es-pam-01] t1-es-gld/Bl0pFprtToeWDUMqi-C6DA
[2015-02-26 18:45:52,137][INFO ][discovery.zen ] [tg-es-pam-01] failed to send join request to master [[tg-es-pam-02][1kmhsjwfRsyyowmG42XnJg][d31804281a9c][inet[/172.17.0.3:9300]]], reason [RemoteTransportException[[tg-es-pam-01][inet[/172.17.0.3:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[tg-es-pam-01][Bl0pFprtToeWDUMqi-C6DA][eafa8f60b3cb][inet[/172.17.0.3:9300]]] not master for join request from [[tg-es-pam-01][Bl0pFprtToeWDUMqi-C6DA][eafa8f60b3cb][inet[/172.17.0.3:9300]]]]; ], tried [3] times
Errors on VM 2
[2015-02-26 18:57:47,148][WARN ][discovery.zen ] [tg-es-pam-02] failed to connect to master [[tg-es-pam-01][Bl0pFprtToeWDUMqi-C6DA][eafa8f60b3cb][inet[/172.17.0.3:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [tg-es-pam-01][inet[/172.17.0.3:9300]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:807)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:741)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:714)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:150)
at org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:441)
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:393<nabble_embed></nabble_embed>)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$6000(ZenDiscovery.java:80)
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1320)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.netty.channel.ConnectTimeoutException: connection timed out: /172.17.0.3:9300
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:139)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)