Hi.
We have this setup:
- 4-nodes Elasticsearch cluster on dedicated physical server (only 3 of them has master role)
- Kubernetes cluster, which runs all parts of our application
My idea is to run some client-only nodes inside Kubernetes cluster to speedup everything a little bit The client nodes should act as "load-balancers" and all traffic which goes currently directly from Kubernetes to Elasticsearch should go first through this client nodes.
I am not sure what I am doing wrong, but it looks, that even on the client nodes, the transport port has to be accessible from rest of the cluster (which is not easy to do in case of Kubernetes). Am I right or I have just something wrong in my configuration?
I am using the default image docker.elastic.co/elasticsearch/elasticsearch:6.8.6
With following config:
discovery.zen.ping.unicast.hosts: elasticsearch-node-01.bf.svc:9300,elasticsearch-node-02.bf.svc:9300,elasticsearch-node-03.bf.svc:9300
discovery.zen.minimum_master_nodes: 1
cluster.name: xxx
network.host: 0.0.0.0
node.data: false
node.ingest: false
node.master: false
node.ml: false
cluster.remote.connect: false
And getting this error:
[INFO ][o.e.d.z.ZenDiscovery ] [elasticsearch-client-node-0] failed to send join request to master [{mic665}{hdVeXXXiCj6Q}{eENqgPXXXzVvA}{X.X.X.208}{X.X.X.208:9300}{ml.machine_memory=134595375104, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[mic665][X.X.X.208:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[elasticsearch-client-node-0][10.233.83.19:9300] connect_exception]; nested: IOException[connection timed out: 10.233.83.19/10.233.83.19:9300]; ]
Where X.X.X.208 is IP of the mic665 node and 10.233.83.19 is Pods internal IP in Kubernetes cluster (which is currently not reachable from outside of te K8s cluster).