Cannot get cross cluster search working for Elasticsearch 6.4 on Kubernetes

(Scott Preddy) #1

We cannot get cross cluster search to work on Kubernetes. We cannot use this feature without exposing the remote nodes on the target Kubernetes cluster by way of Hostnetworking (which opens the ip and port of the Elastic search nodes to outside Kubernetes). We cannot use Kubernetes services or Ingresses on remote Elastic nodes because once Elastic search discovery kicks in, the mechanism passes back hostnames and addresses local to the Kubernetes cluster which are unusable. We have found a way to use the gateway attribute to section off a few nodes on remote clusters and enable hostnetworking on them only. This is still not releasable into production. We need a way to assign an Ingress to a set of Elastic search nodes on Kubernetes and point to that ingress from client Elasticsearchs and have no discovery take effect. This is impossible today because one can only specify seed addresses at the client and not usable endpoints. This has our production broken on Kubernetes for Elasticsearch and is a show stopper.