User docker to build elasticsearch cluster on multi-machine

Elastic Stack Elasticsearch
1
  • Machine A runs 3 es nodes , Machine B adds a new data nodes it faield when joining the cluster on A
  • machine A es master node settings:
      cluster.name: es-cluster
      node.name: es-master
      node.master: true
      node.data: false
      network.host: 0.0.0.0
      http.port: 9200
      transport.port: 9300
      discovery.seed_hosts: [ "es-master", "es-slave1", "es-eligible", "10.85.120.33:9300"]
      cluster.initial_master_nodes: ["es-eligible", "es-master"]
      http.cors.enabled: true
      http.cors.allow-origin: "*"
      xpack.security.enabled: false
      #http.cors.allow-headers: "Authorization"
  • Machine B node settings:
      cluster.name: es-cluster
      node.name: es-slave2
      node.master: false
      node.data: true
      network.host: 0.0.0.0
      http.port: 9200
      transport.port: 9300
      discovery.seed_hosts: ['10.85.121.212:9300', "10.85.121.212:9301", "10.85.121.212:9302", "es-slave2"]
      cluster.initial_master_nodes:
      - 10.85.121.212:9300 
      - 10.85.121.212:9302
      http.cors.enabled: true
      http.cors.allow-origin: "*"
      xpack.security.enabled: false
      #http.cors.allow-headers: "Authorization"
2

First, which telnet works? You can telnet from machine 2 to ALL of these IP:port combinations? '10.85.121.212:9300', "10.85.121.212:9301", "10.85.121.212:9302" AND from those to the node on machine 2 (10.85.120.33:9300) ? If that works, then I have no idea, but otherwise:

In Docker, you have to make sure your containers on machine A are accessible from other machines, which is hard by default, depending on your networking.

ES often will not know the IP it can use for other machines to reach it - your 10.85.121.212 - you may need to set that on machine 1 nodes' network.publish_host - else they'll publish their docker IPs which may be something different, like 192.x, etc.

Also, for Machine B, initial master nodes are usually node names, not IPs (though IPs can work), but not matter as you are not forming a new cluster. Suggest removing them as the cluster is formed, and as soon as this 'new' master talks to the current master/slave it'll know the cluster into. Initial master nodes is so a new cluster can setup during bootstrap.

3

telnet works on all containers . I will try to set nodes' network.publish_host . thanks

4

Yeah, I bet the nodes not know their IPs to publish them until you set the publish IP - see
GET /_cat/nodes to see what they think they are.

5

If i set the machine-A's Ip to network.publish_host , then containers will not be able to find the master

6

Welcome do Docker - that is a core challenge - you need to setup for an IP/port that BOTH local containers AND remote containers can reach, and tell the nodes what those are.

If you use default Docker settings and basic port forwarding (e.g. -P 9201:9201), you can use the VM's IP and the ports like 9200, 9201, etc. as both local and remote containers should be able to reach that (though been a while since I set it up).

7

this is how is runs
esconfig

8

Hmm, as far as I know you can't really change ports, as there is no way to change the published port, just the IP.

So if ES is listening 9200/9300 it has to appear that way on the advertised IP, too as that's how the internal list is managed - you can change IPs but not ports from my reading of it.

So change your ES configured port in the container (via YML or env vars) to 9201/9301 and 9202/9302, and map them 9201:9201, etc., and then the ES knows it's on 9201 and knows the IP .212 and will advertise those.

The it just might work. I'm confused why/how your Machine A works now, though as they will be advertising their 9200/9300 ports in the cluster state, too.

Bottom line, change IPs if you want via publish setting, but don't change ports.

Others may know more, of course (and likely).

9

i use docker-compose on A to run the cluster

10

Oh, I was wrong, you can publish the port; set transport.publish_port: 9301, etc. for the ports and then it'll announce correctly, but this might break your local node though not sure as they are using the internal network (see below) though best if you specify IP/port that everyone can reach/see, then it's clear - set both publish IP and port (to 9301, 9302, etc.)

Here are Docker Compose docs, but in this version probably what you are using, the containers are connecting locally on port 9300 inside Docker; if you do this, you'll have the problem you are seeing as the ES servers are not exposed for transport port 9300 outside.

So set ES to listen on 9300, 9301, 9302 and set publish port on each to that, and map 9301:9301, etc. and I bet it'll all work. I hope.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.