Issue with Communication Between Two Elasticsearch Nodes Using Docker

Elastic Search
1

Hello everyone,

I am currently experimenting with Elasticsearch, and I would like to set up a small infrastructure with 2 nodes that are not on the same server but within the same cluster. I’m using Docker for this. While I can successfully start Elasticsearch, the nodes are not communicating with each other (I tried running them on the same server, and it worked). I believe the issue is related to my Docker network configuration.

If anyone has any suggestions or guidance, I’d greatly appreciate it!

Have a great day! :slight_smile:

2

However, I am using two Debian VMs with VirtualBox.

3

Hi @Telos,

Wanted to check first - did you make sure that you have connectivity between hosts, ports are open on the host and docker ports are properly exposed?

4

I am certain that there is a connection between my two hosts. To verify this, I first performed a ping from host 1 to host 2, and vice versa from host 2 to host 1.

To ensure the ports are open, I used Telnet, and it confirmed that the connection works correctly.

As for Docker ports, that’s something I’ll investigate further. I must admit this is the first time I’m trying to connect to a container on a different machine, so I’ll look into this aspect more closely.

edit: After reviewing my Docker Compose file, I believe it is configured correctly. Here it is:
(Master Node)

services:
  elasticsearch-master:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.10.1
    container_name: elasticsearch-master
    environment:
      - discovery.seed_hosts=IP_MACHINE_MASTER,IP_MACHINE_DATA
      - cluster.initial_master_nodes=elasticsearch-master
      - cluster.name=es-cluster
      - node.name=elasticsearch-master
      - node.roles=master
      - network.host=0.0.0.0
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      es-net:
        ipv4_address: 192.168.1.10  # Adresse IP fixe du nœud master
networks:
  es-net:
    driver: bridge
    ipam:
      config:
        - subnet: 192.168.1.0/24

(Data Node)

services:
  elasticsearch-data:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.10.1
    container_name: elasticsearch-data
    environment:
      - discovery.seed_hosts=IP_MACHINE_MASTER,IP_MACHINE_DATA
      - cluster.name=es-cluster
      - node.name=elasticsearch-data
      - node.roles=data
      - network.host=0.0.0.0
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - "9201:9200"
      - "9301:9300"
    networks:
      es-net:
        ipv4_address: 192.168.1.11  # Adresse IP fixe du nœud data
networks:
  es-net:
    driver: bridge
    ipam:
      config:
        - subnet: 192.168.1.0/24
5

Could it be a problem that's solved by changing second docker-compose's ports to 9200:9200 and 9300:9300.

I'm not super familiar with such setup, but might be worth trying?

6

I changed the ports, but it still doesn't work. As shown below, I still have only one node in my cluster (here we are on the VM with the master node):

telos@debian:~$ sudo docker exec -it elasticsearch-master curl -X GET "http://localhost:9200/_cat/nodes?v"
ip           heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.57.2           60          94  31    1.54    0.90     0.46 m         *      elasticsearch-master

Here is the new configuration after changing the ports of my data node configuration (and also the IPs—I had to adjust the host-only settings due to conflicts with other VMs ^^):

services:
  elasticsearch-data:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.10.1
    container_name: elasticsearch-data
    environment:
      - discovery.seed_hosts=192.168.57.2,192.168.57.3
      - cluster.name=es-cluster
      - node.name=elasticsearch-data
      - node.roles=data
      - network.host=0.0.0.0
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.security.enabled=false
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      es-net:
        ipv4_address: 192.168.57.3
networks:
  es-net:
    driver: bridge
    ipam:
      config:
        - subnet: 192.168.57.0/24
7

Hello,

As per the document :
discovery.seed_hosts

(Static) Provides a list of the addresses of the master-eligible nodes in the cluster. May also be a single string containing the addresses separated by commas. Each address has the format host:port or host. The host is either a host name to be resolved by DNS, an IPv4 address, or an IPv6 address. IPv6 addresses must be enclosed in square brackets. If a host name resolves via DNS to multiple addresses, Elasticsearch uses all of them. DNS lookups are subject to JVM DNS caching.

Could you please try :

  • discovery.seed_hosts=["192.168.57.2","192.168.57.3"]

Thanks!!

8

Thank you! But it still doesn't work, I only have one node in the cluster...
I think the issue is mainly related to my Docker network.

9

Thank you all for your help! I managed to get it working by using:
network_mode: "host"

I’m not sure if this is the best approach, so if you have other suggestions, please feel free to share! The downside is that the container is now directly exposed on my network, which defeats the isolation aspect of Docker containers.

1 Like
10

A bridge network is a NAT network, the IP addresses of the containers are private/internal and only routable by the other containers running on that docker host.

To access a container remotely you need to expose the ports as you've done in the docker compose. This sets up port forwarding from the docker host to that specific container on the specified port.

This is why you cannot expose the same port from two containers on the same host, even if the containers each have their own private IP address.

To proceed with a bridge network, you need the discovery node list to contain the IP addresses of each docker host, not the IP addresses of the containers.