I am currently experimenting with Elasticsearch, and I would like to set up a small infrastructure with 2 nodes that are not on the same server but within the same cluster. I’m using Docker for this. While I can successfully start Elasticsearch, the nodes are not communicating with each other (I tried running them on the same server, and it worked). I believe the issue is related to my Docker network configuration.
If anyone has any suggestions or guidance, I’d greatly appreciate it!
I am certain that there is a connection between my two hosts. To verify this, I first performed a ping from host 1 to host 2, and vice versa from host 2 to host 1.
To ensure the ports are open, I used Telnet, and it confirmed that the connection works correctly.
As for Docker ports, that’s something I’ll investigate further. I must admit this is the first time I’m trying to connect to a container on a different machine, so I’ll look into this aspect more closely.
edit: After reviewing my Docker Compose file, I believe it is configured correctly. Here it is:
(Master Node)
I changed the ports, but it still doesn't work. As shown below, I still have only one node in my cluster (here we are on the VM with the master node):
telos@debian:~$ sudo docker exec -it elasticsearch-master curl -X GET "http://localhost:9200/_cat/nodes?v"
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.57.2 60 94 31 1.54 0.90 0.46 m * elasticsearch-master
Here is the new configuration after changing the ports of my data node configuration (and also the IPs—I had to adjust the host-only settings due to conflicts with other VMs ^^):
(Static) Provides a list of the addresses of the master-eligible nodes in the cluster. May also be a single string containing the addresses separated by commas. Each address has the format host:port or host. The host is either a host name to be resolved by DNS, an IPv4 address, or an IPv6 address. IPv6 addresses must be enclosed in square brackets. If a host name resolves via DNS to multiple addresses, Elasticsearch uses all of them. DNS lookups are subject to JVM DNS caching.
Thank you all for your help! I managed to get it working by using: network_mode: "host"
I’m not sure if this is the best approach, so if you have other suggestions, please feel free to share! The downside is that the container is now directly exposed on my network, which defeats the isolation aspect of Docker containers.
A bridge network is a NAT network, the IP addresses of the containers are private/internal and only routable by the other containers running on that docker host.
To access a container remotely you need to expose the ports as you've done in the docker compose. This sets up port forwarding from the docker host to that specific container on the specified port.
This is why you cannot expose the same port from two containers on the same host, even if the containers each have their own private IP address.
To proceed with a bridge network, you need the discovery node list to contain the IP addresses of each docker host, not the IP addresses of the containers.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.