Elasticsearch cluster with docker - handshake failed

[maciejgawlowski]

Hi, I try to connect one elasticsearch data node to another elasticsearch data-master node. Both elasticsearch are running in docker containers on two different virtual machines. When I try to connect I get error:

[2018-06-01T12:21:19,083][WARN ][o.e.d.z.ZenDiscovery ] [elastic2] failed to connect to master [{elastic1}{Cs3L9TgtRBq5nlEnDn31IA}{Qr7ZvpCKTPaEx_gZR2RJLw}{172.20.0.2}{172.20.0.2:9300}], retrying...
org.elasticsearch.transport.ConnectTransportException: [elastic1][172.20.0.2:9300] handshake failed. unexpected remote node {elastic2}{efMUvUCBQIe3cezvDwPSwA}{t-zLPT8LQ7KGEepbyuTxtw}{172.20.0.2}{172.20.0.2:9300}
at org.elasticsearch.transport.TransportService.lambda$connectToNode$3(TransportService.java:335) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:514) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:331) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:318) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:515) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:483) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.discovery.zen.ZenDiscovery.access$2500(ZenDiscovery.java:90) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1253) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.3.jar:6.2.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

I attach configuration:
ELASTICSEARCH DATA-MASTER:

docker-compose:
version: '2'

services:

elasticsearch:
build:
context: elasticsearch/
restart: always
volumes:
- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
#store data as docker volume (stored in /home/docker-data/volumes)
- elastic_search_data:/usr/share/elasticsearch/data
ports:
- "9200:9200"
- "9300:9300"
environment:
- cluster.name=elasticsearch-cluster
- ES_JAVA_OPTS=-Xmx1024m -Xms1024m
networks:
- elk

logstash:
build:
context: logstash/
restart: always
volumes:
- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
- ./logstash/pipeline:/usr/share/logstash/pipeline:ro
- ./logstash/cert:/etc/pki/tls:ro
- ./logstash/patterns:/usr/share/logstash/patterns:ro
ports:
- "5000:5000"
environment:
- LS_JAVA_OPTS=-Xmx512m -Xms512m
networks:
- elk
depends_on:
- elasticsearch

kibana:
build:
context: kibana/
restart: always
volumes:
- ./kibana/config/:/usr/share/kibana/config:ro
ports:
- "5601:5601"
environment:
- KI_JAVA_OPTS=-Xmx512m -Xms512m
networks:
- elk
depends_on:
- elasticsearch

volumes:
elastic_search_data:
driver: local

networks:
elk:
driver: bridge

elasticsearch.yml:
cluster.name: elasticsearch-cluster
network.host: 0.0.0.0
node.name: elastic1
node.master: true
node.data: true

discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts: ["172.28.4.185"]

and second node:
docker-compose:
version: '2'

services:

elasticsearch:
build:
context: elasticsearch/
restart: always
volumes:
- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
#store data as docker volume (stored in /home/docker-data/volumes)
- elastic_search_data:/usr/share/elasticsearch/data
ports:
- "9200:9200"
- "9300:9300"
environment:
- cluster.name=elasticsearch-cluster
- ES_JAVA_OPTS=-Xmx1024m -Xms1024m
networks:
- elk

volumes:
elastic_search_data:
driver: local

networks:
elk:
driver: bridge

elasticsearch.yml:
cluster.name: elasticsearch-cluster
network.host: 0.0.0.0
node.name: elastic2
node.master: false
node.data: true

discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts: ["172.28.4.184"]

Could anyone help me?

[maciejgawlowski]

Iv found a solution for the issue. Ive connected two docker services via docker network with type overlay. Overlay network allows to connect services running on different host machines. You can find how to configure overlay network in docker docs or in this video: https://www.youtube.com/watch?v=nGSNULpHHZc . I attach my current configuration:

Elasticsearch data-master node:
version: '3.6'

services:

elasticsearch-1:
...
networks:
- elasticsearch_network

logstash:
...
networks:
- elasticsearch_network
depends_on:
- elasticsearch-1

kibana:
...
networks:
- elasticsearch_network
depends_on:
- elasticsearch-1

networks:
elasticsearch_network:
driver: overlay
driver_opts:
encrypted: "true"

elasticsearch.yml:
cluster.name: elasticsearch-cluster
network.host: 0.0.0.0
node.name: elasticsearch-1
node.master: true
node.data: true

discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts: [elasticsearch-2]

Elasticsearch only data node:

version: '3.6'

services:

elasticsearch-2:
...
networks:
- elk_elasticsearch_network

networks:
elk_elasticsearch_network:
external: true
driver: overlay

elasticsearch.yml:
cluster.name: elasticsearch-cluster
network.host: 0.0.0.0
node.name: elasticsearch-2
node.master: false
node.data: true

discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts: [elasticsearch-1]

[andersonVSA (Anderson De Vasconcelos Sant'ana)]

Thank you maciejgawlowski!
It worked for me in my 2 aws machines, after I created the docker overlay network with docker-swarm.

[maciejgawlowski]

Great!