Elasticsearch clustering on two hosts using docker-compose

[pastechecker (Charlie)]

Hello.
I am attempting to form a cluster with 6 docker containers on two docker hosts.
Topology below:

Docker-compose.yml on ELK1:

version: '3.7'

services:
elk1_es01:
container_name: elk1_es01
image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
environment:
- node.name=elk1_es01
- discovery.seed_hosts=192.168.1.1:9301,elk1_es01,elk1_es02
- cluster.initial_master_nodes=192.168.1.1:9301,elk1_es01
- "ES_JAVA_OPTS=-Xms31g -Xmx31g"
volumes: ['elk1_data01:/usr/share/elasticsearch/data']
ports:
- 9211:9200
- 9311:9300

elk1_es02:
container_name: elk1_es02
image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
environment:
- node.name=elk1_es02
- discovery.seed_hosts=192.168.1.1:9301,elk1_es01,elk1_es02
- cluster.initial_master_nodes=192.168.1.1:9301,elk1_es01
- "ES_JAVA_OPTS=-Xms31g -Xmx31g"
volumes: ['elk1_data02:/usr/share/elasticsearch/data']
ports:
- 9212:9200
- 9312:9300

elk1_es03:
container_name: elk1_es03
image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
environment:
- node.name=elk1_es03
- discovery.seed_hosts=192.168.1.1:9301,elk1_es01,elk1_es02
- cluster.initial_master_nodes=192.168.1.1:9301,elk1_es01
- "ES_JAVA_OPTS=-Xms31g -Xmx31g"
volumes: ['elk1_data03:/usr/share/elasticsearch/data']
ports:
- 9213:9200
- 9313:9300

volumes: {"elk1_data01", "elk1_data02", "elk1_data03"}

Docker-compose.yml on ELK2:

version: '3.7'

services:
elk2_es01:
container_name: elk2_es01
image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
environment:
- node.name=elk2_es01
- discovery.seed_hosts=192.168.1.2:9311,elk2_es01
- cluster.initial_master_nodes=192.168.1.2:9311,elk2_es01
- "ES_JAVA_OPTS=-Xms31g -Xmx31g"
volumes: ['elk2_data01:/usr/share/elasticsearch/data']
ports:
- 9201:9200
- 9301:9300

elk2_es02:
container_name: elk2_es02
image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
environment:
- node.name=elk2_es02
- discovery.seed_hosts=192.168.1.2:9311,elk2_es01
- cluster.initial_master_nodes=192.168.1.2:9311,elk2_es01
- "ES_JAVA_OPTS=-Xms31g -Xmx31g"
volumes: ['elk2_data02:/usr/share/elasticsearch/data']
ports:
- 9202:9200
- 9302:9300

elk2_es03:
container_name: elk2_es03
image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
environment:
- node.name=elk2_es03
- discovery.seed_hosts=192.168.1.2:9311,elk2_es01
- cluster.initial_master_nodes=192.168.1.2:9311,elk2_es01
- "ES_JAVA_OPTS=-Xms31g -Xmx31g"
volumes: ['elk2_data03:/usr/share/elasticsearch/data']
ports:
- 9203:9200
- 9303:9300

volumes: {"elk2_data01", "elk2_data02", "elk2_data03"}

curl http://192.168.1.2:9211/_cat/nodes
172.22.0.2 4 94 18 7.70 5.35 10.72 mdi - elk1_es03
172.22.0.4 3 94 18 7.70 5.35 10.72 mdi - elk1_es01
172.22.0.3 2 94 18 7.70 5.35 10.72 mdi * elk1_es02

curl http://192.168.1.1:9201/_cat/nodes
192.168.192.4 4 96 24 7.76 8.82 8.78 mdi * elk2_es01
192.168.192.2 2 96 24 7.76 8.82 8.78 mdi - elk2_es03
192.168.192.3 3 96 24 7.76 8.82 8.78 mdi - elk2_es02

I also see that they are trying to talk with the success on the data ports:

09:47:08.154752 IP 192.168.1.2.37574 > 192.168.1.1.9301: Flags [S], seq 3973389975, win 29200, options [mss 1460,sackOK,TS val 3699807186 ecr 0,nop,wscale 7], length 0
09:47:08.154869 IP 192.168.1.1.9301 > 192.168.1.2.37574: Flags [S.], seq 2762589956, ack 3973389976, win 28960, options [mss 1460,sackOK,TS val 646457207 ecr 3699807186,nop,wscale 7], length 0
09:47:08.154879 IP 192.168.1.1.9301 > 192.168.1.2.37574: Flags [S.], seq 2762589956, ack 3973389976, win 28960, options [mss 1460,sackOK,TS val 646457207 ecr 3699807186,nop,wscale 7], length 0
09:47:08.157545 IP 192.168.1.2.37574 > 192.168.1.1.9301: Flags [.], ack 1, win 229, options [nop,nop,TS val 3699807188 ecr 646457207], length 0
09:47:08.474771 IP 192.168.1.2.37574 > 192.168.1.1.9301: Flags [P.], seq 1:376, ack 1, win 229, options [nop,nop,TS val 3699807508 ecr 646457207], length 375
09:47:08.474967 IP 192.168.1.1.9301 > 192.168.1.2.37574: Flags [.], ack 376, win 235, options [nop,nop,TS val 646457527 ecr 3699807508], length 0
09:47:08.474975 IP 192.168.1.1.9301 > 192.168.1.2.37574: Flags [.], ack 376, win 235, options [nop,nop,TS val 646457527 ecr 3699807508], length 0
09:47:08.845807 IP 192.168.1.1.9301 > 192.168.1.2.37574: Flags [P.], seq 1:1501, ack 376, win 235, options [nop,nop,TS val 646457898 ecr 3699807508], length 1500
09:47:08.845818 IP 192.168.1.1.9301 > 192.168.1.2.37574: Flags [P.], seq 1:1501, ack 376, win 235, options [nop,nop,TS val 646457898 ecr 3699807508], length 1500
09:47:08.846216 IP 192.168.1.2.37574 > 192.168.1.1.9301: Flags [.], ack 1501, win 252, options [nop,nop,TS val 3699807879 ecr 646457898], length 0
09:47:09.476132 IP 192.168.1.2.37574 > 192.168.1.1.9301: Flags [P.], seq 376:1678, ack 1501, win 252, options [nop,nop,TS val 3699808509 ecr 646457898], length 1302
09:47:09.476249 IP 192.168.1.1.9301 > 192.168.1.2.37574: Flags [.], ack 1678, win 258, options [nop,nop,TS val 646458528 ecr 3699808509], length 0

However, I do not see the cluster working together with elk1 and elk2. The direct connection is over fiber optic bundle addressed 192.168.1.1 <-> 192.168.1.2.
I am using the default bridge network in the docker-compose.

No logs are being generated in the cluster forming between elk1 and elk2, so I must be missing some configuration.

What am I missing here?

Regards

[Julien]

Can you check all the host are in the same docker network by using ping from one container to the container on the other docker host:
docker exec -it elk1_es01 /bin/bash
$ ping elk2_es01

If you cannot ping then the containers are running in 2 separate docker networks, this would not be related to elasticsearch but just making sure you have the container being accessible over network

I am no expert on docker but if you use "docker-compose up -d" (I guess this is what you use because container names relates to this feature as opposed to docker swarm) to use default bridge network driver, this will just create a network valid on one docker host so it's expected you cannot connect to a separate docker network only known to another docker host. You probably need to look at docker swarm and overlay network (https://docs.docker.com/network/overlay/)

[pastechecker (Charlie)]

For the nodes on the other network I have used not named container, but the IP.

root@0d8a6709b8b6 elasticsearch]# ping 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=63 time=0.344 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=63 time=0.279 ms
64 bytes from 192.168.1.2: icmp_seq=5 ttl=63 time=0.330 ms
64 bytes from 192.168.1.2: icmp_seq=7 ttl=63 time=0.252 ms
64 bytes from 192.168.1.2: icmp_seq=9 ttl=63 time=0.310 ms
64 bytes from 192.168.1.2: icmp_seq=11 ttl=63 time=0.287 ms
64 bytes from 192.168.1.2: icmp_seq=13 ttl=63 time=0.264 ms
64 bytes from 192.168.1.2: icmp_seq=15 ttl=63 time=0.250 ms
64 bytes from 192.168.1.2: icmp_seq=17 ttl=63 time=0.292 ms

[root@0d8a6709b8b6 ca]# curl --cacert ca.crt -u elastic:PleaseChangeMe https://192.168.1.2:9211/_cat/nodes
192.168.48.3 5 97 14 2.49 7.18 12.96 mdi * elk1_es03
192.168.48.2 5 97 14 2.49 7.18 12.96 mdi - elk1_es02
192.168.48.4 6 97 14 2.49 7.18 12.96 mdi - elk1_es01

So I do have a connection over TCP and ICMP.