# Elasticsearch cluster across multiple docker hosts **URL:** \ **Category:** Elasticsearch\ **Tags:** docker\ **Created:** [April 26, 2024, 7:30pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-across-multiple-docker-hosts/358314 "2024-04-26T19:30:08Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Janusz\_Mikluszka](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/janusz_mikluszka/32/122075_2.png) [@Janusz\_Mikluszka](https://discuss.elastic.co/u/Janusz_Mikluszka)\ **Post date:** [April 26, 2024, 7:30pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-across-multiple-docker-hosts/358314/1 "2024-04-26T19:30:09Z")
Hello, I'm trying to create multi node cluster across multiple VMs running docker. For now I tried setting up master node on my local docker engine. When second node is in the same docker network then there is no issue with connecting them. elasticsearch.yml of the master node ```auto xpack.security.enabled: true cluster.name: my-elastic-cluster cluster.initial_master_nodes: [es01] node.name: es01 transport.host: 0.0.0.0 xpack.security.http.ssl.key: certs/es01/es01.key xpack.security.http.ssl.certificate: certs/es01/es01.crt xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.key: certs/es01/es01.key xpack.security.transport.ssl.certificate: certs/es01/es01.crt xpack.security.transport.ssl.certificate_authorities: certs/ca/ca.crt xpack.security.transport.ssl.verification_mode: certificate ``` Elasticsearch.yml of the second node ```auto xpack.security.enabled: true cluster.name: my-elastic-cluster cluster.initial_master_nodes: [es01] discovery.seed_hosts: - es01 node.name: es02 transport.host: es02 node.roles: - transform - data_frozen - remote_cluster_client - data - ml - data_content - data_hot - data_warm - data_cold - ingest xpack.security.http.ssl.key: certs/es02/es02.key xpack.security.http.ssl.certificate: certs/es02/es02.crt xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.key: certs/es02/es02.key xpack.security.transport.ssl.certificate: certs/es02/es02.crt xpack.security.transport.ssl.certificate_authorities: certs/ca/ca.crt xpack.security.transport.ssl.verification_mode: certificate ``` And the docker-compose ```auto services: es01: image: docker.elastic.co/elasticsearch/elasticsearch:7.17.10 ports: - "0.0.0.0:9200:9200" - "0.0.0.0:9300:9300" environment: - "http.host=0.0.0.0" - "ES_JAVA_OPTS=-Xms512m -Xmx512m" - ELASTIC_PASSWORD=somepass volumes: - ./elasticsearch/config/es01/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml - elastic-data:/usr/share/elasticsearch/data - ./certs:/usr/share/elasticsearch/config/certs extra_hosts: - "host.docker.internal:host-gateway" - "es03:host-gateway" es02: image: docker.elastic.co/elasticsearch/elasticsearch:7.17.10 expose: - 9200 - 9300 environment: - "http.host=0.0.0.0" - "ES_JAVA_OPTS=-Xms2g -Xmx2g" - ELASTIC_PASSWORD=somepass volumes: - ./elasticsearch/config/es02/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml - elastic-data:/usr/share/elasticsearch/data - ./certs:/usr/share/elasticsearch/config/certs extra_hosts: - "host.docker.internal:host-gateway" - "es03:host-gateway" ``` For 3rd node ```auto xpack.security.enabled: true cluster.name: my-elastic-cluster cluster.initial_master_nodes: [es01] discovery.seed_hosts: - es01:9302 node.name: es03 node.master: false transport.port: 9300 transport.host: 0.0.0.0 xpack.security.http.ssl.key: certs/es03/es03.key xpack.security.http.ssl.certificate: certs/es03/es03.crt xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.key: certs/es03/es03.key xpack.security.transport.ssl.certificate: certs/es03/es03.crt xpack.security.transport.ssl.certificate_authorities: certs/ca/ca.crt xpack.security.transport.ssl.verification_mode: certificate ``` and it's docker compose ```auto services: es02: image: docker.elastic.co/elasticsearch/elasticsearch:7.17.10 ports: - "0.0.0.0:19200:9200" - "0.0.0.0:19300:9300" environment: - "http.host=0.0.0.0" - "ES_JAVA_OPTS=-Xms1g -Xmx1g" - ELASTIC_PASSWORD=somepass volumes: - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml - elastic-data:/usr/share/elasticsearch/data - ./certs:/usr/share/elasticsearch/config/certs restart: unless-stopped extra_hosts: - "host.docker.internal:host-gateway" - "es01:host-gateway" ``` I used VM to which I created SSH Tunnel with command `ssh my-vm -L 0:19300:127.0.0.1:19300 -R 0:9302:127.0.0.1:9300`. To explain the tunnel: - -L listens on my local 19300 port and forwards it to VMs 19300 port. - -R forwards traffic from VM 9302 port to my local 9300 port which is used by Elasticsearch master node es01 container has IP 172.23.0.3 (assigned by docker) and the es03 has IP 172.18.0.2. They are on different hosts. es03 is able to connect with es01 (msater node) but then I guess es01 sends it's `publish_host` address and port and the connection fails Whole message ```auto [connectToRemoteMasterNode[172.18.0.1:19300]] completed handshake with [{es01}{tigqAqNMRE6L4GNKXZF9lw}{OLUAlRl7TwmL_zwD96FCxA}{172.23.0.3}{172.23.0.3:9300}{cdfhilmrstw}{ml.machine_memory=33359724544, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=2147483648, transform.node=true}] but followup connection failed ``` The 172.18.0.1 is gateway address so basically address of host. Can I just set the correct IP:port of the master node that es03 can use to communicate with it? Or maybe I have to use some iptables? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 26, 2024, 8:41pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-across-multiple-docker-hosts/358314/2 "2024-04-26T20:41:04Z")
Hi Janusz and welcome! I'd suggest looking at [these docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html), particularly the section on [binding and publishing](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#modules-network-binding-publishing) to understand better how Elasticsearch needs its network to be configured. Basically it looks like your nodes are not accessible at their transport publish addresses. It can be tricky to get everything lined up correctly in Docker since it all depends on exactly how your environment is configured. In particular the `followup connection failed` message indicates that node `es01` is accessible at `172.18.0.1:19300`, but it thinks its address is `172.23.0.3:9300` and it is not accessible at this address. You either need to make it accessible at that address, or else adjust its publish address to one at which it is accessible. ---
**Author:** ![Janusz\_Mikluszka](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/janusz_mikluszka/32/122075_2.png) [@Janusz\_Mikluszka](https://discuss.elastic.co/u/Janusz_Mikluszka)\ **Post date:** [April 28, 2024, 9:44pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-across-multiple-docker-hosts/358314/3 "2024-04-28T21:44:05Z")
Thank you for your answer. So if I understand correctly master node has to be accessible to every node through it's publish\_host. If 1 node is behind some port forwarding or NAT then it will fail to connect to master node. Let me confirm 1 more thing. Do I correctly understand that even though we can set multiple addresses to `transport.publish_host` only 1 will be chosen by master node and every node should be able to connect to master through this address, yes? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 29, 2024, 4:54am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-across-multiple-docker-hosts/358314/4 "2024-04-29T04:54:33Z")
> [@Janusz\_Mikluszka](#): > > So if I understand correctly master node has to be accessible to every node through it's publish\_host. Not just the master node. All nodes have to be accessible _to all other nodes_ at their transport publish addresses. > [@Janusz\_Mikluszka](#): > > even though we can set multiple addresses to `transport.publish_host` only 1 will be chosen by master node It's not the master node which chooses it, but yes Elasticsearch will choose one address to be its publish address. See the note in [these docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#advanced-network-settings), particularly > Ensure each node is accessible at all possible publish addresses It's almost certainly a terrible idea to specify more than one publish address. At best it'll be very confusing, but normally it leads to unrepeatable and very hard-to-diagnose issues. ---
**Author:** ![Janusz\_Mikluszka](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/janusz_mikluszka/32/122075_2.png) [@Janusz\_Mikluszka](https://discuss.elastic.co/u/Janusz_Mikluszka)\ **Post date:** [April 29, 2024, 5:59pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-across-multiple-docker-hosts/358314/5 "2024-04-29T17:59:13Z")
Thank you for reply