# ECK clusters and client sniffing

**URL:** https://discuss.elastic.co/t/eck-clusters-and-client-sniffing/244972
**Category:** Elastic Cloud on Kubernetes (ECK)
**Created:** [August 14, 2020, 8:14am UTC](https://discuss.elastic.co/t/eck-clusters-and-client-sniffing/244972 "2020-08-14T08:14:43Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![Presence](https://avatars.discourse-cdn.com/v4/letter/p/7feea3/32.png) [@Presence](https://discuss.elastic.co/u/Presence)
#### Post date: [August 14, 2020, 8:14am UTC](https://discuss.elastic.co/t/eck-clusters-and-client-sniffing/244972/1 "2020-08-14T08:14:43Z")

</div>

It seems difficult to set up an ECK ElasticSearch cluster where client sniffing works. Does anybody have any suggestions on best approaches here? Our clusters are currently on AWS EKS in 3 availability zones behind a single load balancer with cross-zone load balancing enabled. We seem to suffer from slow detection when nodes fail and I don't think there's any retry on connection failure (can't recall if that's provided by any of the clients anyway? We use the Python client FWIW).

---

<div class="post-metadata">

### Author: ![michael.morello](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael.morello/32/47448_2.png) [@michael.morello](https://discuss.elastic.co/u/michael.morello)
#### Post date: [August 14, 2020, 2:33pm UTC](https://discuss.elastic.co/t/eck-clusters-and-client-sniffing/244972/2 "2020-08-14T14:33:40Z")

</div>

Hi,

There is an ongoing issue to fully support client sniffing with eck: [https://github.com/elastic/cloud-on-k8s/issues/3182](https://github.com/elastic/cloud-on-k8s/issues/3182)

Could you share your Elasticsearch manifest and the client configuration (loadbalancer used, sniffer\_timeout, tls settings...) ?

---

<div class="post-metadata">

### Author: ![jgilly](https://avatars.discourse-cdn.com/v4/letter/j/a88e4f/32.png) [@jgilly](https://discuss.elastic.co/u/jgilly)
#### Post date: [September 7, 2021, 8:00pm UTC](https://discuss.elastic.co/t/eck-clusters-and-client-sniffing/244972/3 "2021-09-07T20:00:31Z")

</div>

My question is similar to the original poster's, what is the best practice to connect to ES on kubernetes with rest clients?

- We are running Elasticsearch version 7.4.2 on kubernetes, AWS EKS, deployed via the elasticsearch  
helm chart, version: 8.0.0-SNAPSHOT, sources: [https://github.com/elastic/elasticsearch](https://github.com/elastic/elasticsearch)
- We're using the Java RestHighLevelClient to query ES.
- Our ES cluster has 3 data nodes and 1 dedicated master in dev. 3 dedicated masters in prod

Our current approach is:

- A dedicated rest client for sniffing, this always sniffs through the load balancer
  - E.g. hit [http://elasticsearch.io:9200/\_nodes/http](http://elasticsearch.io:9200/_nodes/http). save the data node "publish\_address" for the nodes.
  - example /\_nodes response:

```auto
jyLP4JCyQnuq4BvvnSysSA": {
"name": "elasticsearch-data-1",
"transport_address": "10.20.3.140:9300",
"host": "10.20.3.140",
"ip": "10.20.3.140",
"version": "7.4.2",
"build_flavor": "default",
"build_type": "docker",
"build_hash": "2f90bbf7b93631e52bafb59b3b049cb44ec25e96",
"roles": [
"ingest",
"data"
],
"attributes": {
"xpack.installed": "true"
},
"http": {
"bound_address": [
"0.0.0.0:9200"
],
"publish_address": "10.20.3.140:9200",
"max_content_length_in_bytes": 104857600
}
},

```

- the load balancer is a classic lb, and it points only to the data nodes.
- Use the Sniffer with our search and indexing rest clients
- set the search and indexing rest clients NodeSelector to SKIP\_DEDICATED\_MASTERS
- Sniff interval: 4 min
- Sniff on fail delay: 1 min

Thus, every 4 min the sniffer "sniffs" through the load balancer and sets the search or index client's nodes to the es-data pod IP addresses. If a connection to one of the pod ip addresses fails, then it will sniff for new nodes 1 minute later. If the search or index clients have a failing request, they will retry it on the other nodes.

This ~should~ keep us covered during rolling deploys or a full cluster outage. However; is there a different way that is "best practice"? Any comments / concerns with the approach we have now?

Any experience the community can share would be greatly appreciated.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 4, 2022, 8:10am UTC](https://discuss.elastic.co/t/eck-clusters-and-client-sniffing/244972/4 "2022-11-04T08:10:05Z")

</div>


