ES 1.7.3 Docker Swarm publish_host binding to wrong address

[tapuzzo (Tony Apuzzo)]

Not too sure if anyone will have anything for this, but I'm trying to get ES 1.7.3 running in a single-node cluster with Activiti (workflow engine) and am having trouble. The vendor (Alfresco) doesn't yet support a newer version of ElasticSearch.

After a bunch of troubleshooting, I've narrowed the problem down to the fact that when ES comes up, it seems to pick the "wrong" IP address to bind to, even when I specify publish_host.

If I use ES 5.6.2, it does pick the correct IP, but that is not a functional config for Activiti.

I've built a trivial docker-compose file that shows the problem without any 3rd part issues.

version: '3.3'

services:
  es1:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.2
    environment:
      - "network.bind_host=0.0.0.0"
      - "network.publish_host=es1"
      - "discovery.zen.ping.unicast.hosts=es1"
    ports:
      - 9201:9200

  es2:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.2
    environment:
      - "network.bind_host=0.0.0.0"
      - "network.publish_host=es2"
      - "discovery.zen.ping.unicast.hosts=es1"
    ports:
      - 9202:9200

This works with docker-compose as well as docker stack deploy and correctly sets up a 2 node cluster.

If I use the EXACT same config with elasticsearch:1.7.3, the cluster fails to form since the nodes cannot mutually confer (usually). This boils down to the 1.7.3 binding to a 10.255.0.x IP instead of the offered 10.0.0.x IP that swarm assigns. One out of 20 launches or so I get lucky and somehow ES will pick 10.0.0.x for the IP and then the cluster will form.

Using ES5, it always seems to bind to the correct overlay port of 10.0.0.x.

I know this is a longshot but does anyone have suggestions for ways to force the older elasticsearch to pick the right IP address? I tried a bunch of things but haven't been able to get it to consistently work. Even with pre-defined networks it seems that ES will usually somehow pick the wrong IP.

To make it all even stranger, if I use docker-compose to launch the config, then it always works. Part of this is probably that the multicast works with docker-compose but not with docker swarm. But I'm forcing the issue with unicast and it doesn't help.