It seems difficult to set up an ECK ElasticSearch cluster where client sniffing works. Does anybody have any suggestions on best approaches here? Our clusters are currently on AWS EKS in 3 availability zones behind a single load balancer with cross-zone load balancing enabled. We seem to suffer from slow detection when nodes fail and I don't think there's any retry on connection failure (can't recall if that's provided by any of the clients anyway? We use the Python client FWIW).
There is an ongoing issue to fully support client sniffing with eck: https://github.com/elastic/cloud-on-k8s/issues/3182
Could you share your Elasticsearch manifest and the client configuration (loadbalancer used, sniffer_timeout, tls settings...) ?