Elasticsearch, docker and iptables

Elastic Stack Elasticsearch
1

Hello,

Just installed elasticsearch container: https://hub.docker.com/_/elasticsearch/

On 4 RedHat entr Linux 7 VMs.... with a config like so:

cluster.name: Aves
node.name: "aves-01"
path.logs: /var/log/elasticsearch
network.publish_host: 192.168.88.145
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.timeout: 3s
discovery.zen.minimum_master_nodes: 3
discovery.zen.ping.unicast.hosts: [
"192.168.88.190:9300",
"192.168.88.191:9300",
"192.168.88.215:9300",

  ]

When I start the containers... it seems to work fine (kopf plugin shows the cluster state as being good), but, all instances constantly log network connection errors like so:

[2015-08-22 17:44:20,006][WARN ][transport.netty          ] [aves-01] exception caught on transport layer [[id: 0xfda51f31]], closing connection
java.net.NoRouteToHostException: No route to host
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
        at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
        at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

and tcpdump shows constant reject like so:

listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
13:58:52.095383 IP 192.168.88.145 > 172.17.0.2: ICMP host 192.168.88.145 unreachable - admin prohibited, length 68
13:58:52.095387 IP 192.168.88.145 > 172.17.0.2: ICMP host 192.168.88.145 unreachable - admin prohibited, length 68
13:58:52.095621 IP 192.168.88.145 > 172.17.0.2: ICMP host 192.168.88.145 unreachable - admin prohibited, length 68
13:58:52.095623 IP 192.168.88.145 > 172.17.0.2: ICMP host 192.168.88.145 unreachable - admin prohibited, length 68

As if iptables is rejecting traffic from the container to itself? 192.168.88.145 being the host IP, and 172.17.0.2 the container IP. the icmp packet details show this is for tcp port 9300.

Whats going on?

Thank you so much.

2

This is a docker issue, not an Elasticsearch one. From what I have seen this sort of this is pretty common for any docker container.