# Docker Swarm - Internode Communication Issue **URL:** \ **Category:** Elasticsearch\ **Tags:** docker\ **Created:** [January 12, 2022, 2:19pm UTC](https://discuss.elastic.co/t/docker-swarm-internode-communication-issue/294176 "2022-01-12T14:19:05Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Darryl\_Idle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darryl_idle/32/99485_2.png) [@Darryl\_Idle](https://discuss.elastic.co/u/Darryl_Idle)\ **Post date:** [January 12, 2022, 2:19pm UTC](https://discuss.elastic.co/t/docker-swarm-internode-communication-issue/294176/1 "2022-01-12T14:19:05Z")
Hi, My environment for ELK is a Docker Swarm setup. Three containers on three individual hosts to utilize distributed storage. I'm using Swarmpit as my orchestrator to compose the stack as follows: ```auto version: '3.3' services: elasticsearch: image: elasticsearch:7.16.2 environment: ELASTIC_PASSWORD: [password] cluster.initial_master_nodes: elasticsearch cluster.max_shards_per_node: '1500' cluster.name: cisd-prod discovery.seed_hosts: 172.16.14.10,172.16.14.11,172.16.14.12 network.host: 0.0.0.0 network.publish_host: 172.16.14.10 node.name: elasticsearch transport.port: '9300' xpack.monitoring.collection.enabled: 'true' xpack.security.enabled: 'true' xpack.security.http.ssl.certificate: wildcard.crt xpack.security.http.ssl.client_authentication: optional xpack.security.http.ssl.enabled: 'true' xpack.security.http.ssl.key: wildcard.key xpack.security.transport.ssl.client_authentication: required xpack.security.transport.ssl.enabled: 'true' xpack.security.transport.ssl.keystore.path: elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: elastic-certificates.p12 xpack.security.transport.ssl.verification_mode: certificate ports: - 9200:9200 - 9300:9300 volumes: - elastic-swarm:/usr/share/elasticsearch/data - elastic-swarm-config:/usr/share/elasticsearch/config networks: - swarmpit_net logging: driver: json-file deploy: mode: global placement: constraints: - node.labels.elastic_host == true - node.hostname == docker-swarm1 elasticsearch2: image: elasticsearch:7.16.2 environment: ELASTIC_PASSWORD: [password] cluster.initial_master_nodes: elasticsearch cluster.max_shards_per_node: '1500' cluster.name: cisd-prod discovery.seed_hosts: 172.16.14.10,172.16.14.11,172.16.14.12 network.host: 0.0.0.0 network.publish_host: 172.16.14.11 node.name: elasticsearch2 xpack.monitoring.collection.enabled: 'true' xpack.security.enabled: 'true' xpack.security.http.ssl.certificate: wildcard.crt xpack.security.http.ssl.client_authentication: optional xpack.security.http.ssl.enabled: 'true' xpack.security.http.ssl.key: wildcard.key xpack.security.transport.ssl.client_authentication: required xpack.security.transport.ssl.enabled: 'true' xpack.security.transport.ssl.keystore.path: elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: elastic-certificates.p12 xpack.security.transport.ssl.verification_mode: certificate ports: - 9200:9200 - 9300:9300 volumes: - elastic-swarm:/usr/share/elasticsearch/data - elastic-swarm-config:/usr/share/elasticsearch/config networks: - swarmpit_net logging: driver: json-file deploy: mode: global placement: constraints: - node.labels.elastic_host == true - node.hostname == docker-swarm2 elasticsearch3: image: elasticsearch:7.16.2 environment: ELASTIC_PASSWORD: [password] cluster.initial_master_nodes: elasticsearch cluster.max_shards_per_node: '1500' cluster.name: cisd-prod discovery.seed_hosts: 172.16.14.10,172.16.14.11,172.16.14.12 network.host: 0.0.0.0 network.publish_host: 172.16.14.12 node.name: elasticsearch3 xpack.monitoring.collection.enabled: 'true' xpack.security.enabled: 'true' xpack.security.http.ssl.certificate: wildcard.crt xpack.security.http.ssl.client_authentication: optional xpack.security.http.ssl.enabled: 'true' xpack.security.http.ssl.key: wildcard.key xpack.security.transport.ssl.client_authentication: required xpack.security.transport.ssl.enabled: 'true' xpack.security.transport.ssl.keystore.path: elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: elastic-certificates.p12 xpack.security.transport.ssl.verification_mode: certificate ports: - 9200:9200 - 9300:9300 volumes: - elastic-swarm:/usr/share/elasticsearch/data - elastic-swarm-config:/usr/share/elasticsearch/config networks: - swarmpit_net logging: driver: json-file deploy: mode: global placement: constraints: - node.labels.elastic_host == true - node.hostname == docker-swarm3 kibana-swarm: image: kibana:7.16.2 environment: ELASTICSEARCH_HOSTS: https://elasticsearch:9200 ELASTICSEARCH_PASSWORD: [password] ELASTICSEARCH_SSL_CERTIFICATE: /usr/share/kibana/config/wildcard.crt ELASTICSEARCH_SSL_KEY: /usr/share/kibana/config/wildcard.key ELASTICSEARCH_SSL_VERIFICATIONMODE: none ELASTICSEARCH_USERNAME: kibana_system SERVER_MAXPAYLOADBYTES: '20971520' SERVER_SSL_CERTIFICATE: /usr/share/kibana/config/wildcard.crt SERVER_SSL_ENABLED: 'true' SERVER_SSL_KEY: /usr/share/kibana/config/wildcard.key XPACK_SECURITY_HTTP_SSL_CLIENT_AUTHENTICATION: optional ports: - 5601:5601 volumes: - kibana-swarm:/usr/share/kibana/config networks: - swarmpit_net logging: driver: json-file deploy: mode: global placement: constraints: - node.hostname == docker-swarm1 networks: swarmpit_net: external: true volumes: elastic-swarm: external: true elastic-swarm-config: external: true kibana-swarm: external: true ``` For the life of me I can't get communication on port 9300 so the nodes will discover each other. The ports are host-mapped and network.publish\_host is set to their host IPs so they will be static but to no avail. Is there anyone out there that has encountered this kind of setup? Thanks. ---
**Author:** ![Darryl\_Idle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darryl_idle/32/99485_2.png) [@Darryl\_Idle](https://discuss.elastic.co/u/Darryl_Idle)\ **Post date:** [January 12, 2022, 2:23pm UTC](https://discuss.elastic.co/t/docker-swarm-internode-communication-issue/294176/2 "2022-01-12T14:23:07Z")
Getting log lines like: > {"type": "server", "timestamp": "2022-01-12T14:18:29,624Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "cisd-prod", "node.name": "elasticsearch3", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elasticsearch] to bootstrap a cluster: have discovered [{elasticsearch3}{1OG7DlCMShaun8EZGeH21w}{U8ME7A54RaGYZiSz0VZn2g}{172.16.14.12}{172.16.14.12:9300}{cdfhilmrstw}]; discovery will continue using [172.16.14.10:9300, 172.16.14.11:9300] from hosts providers and [{elasticsearch3}{1OG7DlCMShaun8EZGeH21w}{U8ME7A54RaGYZiSz0VZn2g}{172.16.14.12}{172.16.14.12:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" } ---
**Author:** ![Darryl\_Idle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darryl_idle/32/99485_2.png) [@Darryl\_Idle](https://discuss.elastic.co/u/Darryl_Idle)\ **Post date:** [January 12, 2022, 4:31pm UTC](https://discuss.elastic.co/t/docker-swarm-internode-communication-issue/294176/3 "2022-01-12T16:31:14Z")
Solved by wiping the data folder of the 2nd and 3rd node as they already decided to form their own clusters and didn't want to join the 1st node's. Also opened port **7946/tcp** and **4789/udp** in ufw on each node as suggested at: [Manage swarm service networks | Docker Documentation](https://docs.docker.com/engine/swarm/networking/#firewall-considerations) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 9, 2022, 4:32pm UTC](https://discuss.elastic.co/t/docker-swarm-internode-communication-issue/294176/4 "2022-02-09T16:32:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.