# Elasticsearch on Kubernetes losing Cluster Connection every hour

**URL:** <https://discuss.elastic.co/t/elasticsearch-on-kubernetes-losing-cluster-connection-every-hour/237086>\
**Category:** Elasticsearch\
**Created:** [June 15, 2020, 7:58am UTC](https://discuss.elastic.co/t/elasticsearch-on-kubernetes-losing-cluster-connection-every-hour/237086 "2020-06-15T07:58:00Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![fewagewasd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fewagewasd/32/45707_2.png) [@fewagewasd](https://discuss.elastic.co/u/fewagewasd)\
**Post date:** [June 15, 2020, 7:58am UTC](https://discuss.elastic.co/t/elasticsearch-on-kubernetes-losing-cluster-connection-every-hour/237086/1 "2020-06-15T07:58:01Z")

</div>

Hi,

I am migrating an older application using elasticsearch 2.4 (I know that's a very old version, but I'm not able to upgrade at this point) to Kubernetes. I set it up using a StatefulSet and a headless service, which basically works fine.  
However, the connection between cluster nodes is lost after _exactly_ one hour, reestablished, and then lost after an hour again.

There are no errors in the logs, just the messages from nodes joining and leaving the cluster:

```auto
[2020-06-15 04:11:21,652][INFO][discovery.zen] [Midas] master_left [{Isis}{2UKej6PQRya4WnEARlEZwA}{10.100.5.234}{10.100.5.234:9300}], reason [transport disconnected]
[2020-06-15 04:11:21,652][WARN][discovery.zen] [Midas] master left (reason = transport disconnected), current nodes: {{Stunner}{QNE6jkMyR8KqVmRLwbIWhA}{10.100.4.254}{10.100.4.254:9300},{Midas}{66GU3k9BRqGQ2PRAAxnfmQ}{10.100.3.243}{10.100.3.243:9300},}
[2020-06-15 04:11:21,652][INFO][cluster.service] [Midas] removed {{Isis}{2UKej6PQRya4WnEARlEZwA}{10.100.5.234}{10.100.5.234:9300},}, reason: zen-disco-master_failed ({Isis}{2UKej6PQRya4WnEARlEZwA}{10.100.5.234}{10.100.5.234:9300})
[2020-06-15 04:11:51,676][INFO][cluster.service] [Midas] detected_master {Isis}{2UKej6PQRya4WnEARlEZwA}{10.100.5.234}{10.100.5.234:9300}, added {{Isis}{2UKej6PQRya4WnEARlEZwA}{10.100.5.234}{10.100.5.234:9300},}, reason: zen-disco-receive(from master [{Isis}{2UKej6PQRya4WnEARlEZwA}{10.100.5.234}{10.100.5.234:9300}])
[2020-06-15 04:13:20,776][INFO][cluster.service] [Midas] removed {{Stunner}{QNE6jkMyR8KqVmRLwbIWhA}{10.100.4.254}{10.100.4.254:9300},}, reason: zen-disco-receive(from master [{Isis}{2UKej6PQRya4WnEARlEZwA}{10.100.5.234}{10.100.5.234:9300}])
[2020-06-15 04:13:50,794][INFO][cluster.service] [Midas] added {{Stunner}{QNE6jkMyR8KqVmRLwbIWhA}{10.100.4.254}{10.100.4.254:9300},}, reason: zen-disco-receive(from master [{Isis}{2UKej6PQRya4WnEARlEZwA}{10.100.5.234}{10.100.5.234:9300}])
[2020-06-15 05:11:51,660][INFO][discovery.zen] [Midas] master_left [{Isis}{2UKej6PQRya4WnEARlEZwA}{10.100.5.234}{10.100.5.234:9300}], reason [transport disconnected]
[2020-06-15 05:11:51,660][WARN][discovery.zen] [Midas] master left (reason = transport disconnected), current nodes: {{Stunner}{QNE6jkMyR8KqVmRLwbIWhA}{10.100.4.254}{10.100.4.254:9300},{Midas}{66GU3k9BRqGQ2PRAAxnfmQ}{10.100.3.243}{10.100.3.243:9300},}

```

I have found some other threads suggesting to reduce the tcp keepalive settings on the nodes, unfortunately, that didn't help in my case.

Does anyone know what causes this and/or what I can do to fix this?

Thanks!

---

<div class="post-metadata">

**Author:** ![fewagewasd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fewagewasd/32/45707_2.png) [@fewagewasd](https://discuss.elastic.co/u/fewagewasd)\
**Post date:** [June 16, 2020, 8:23am UTC](https://discuss.elastic.co/t/elasticsearch-on-kubernetes-losing-cluster-connection-every-hour/237086/2 "2020-06-16T08:23:47Z")

</div>

Turns out Istio was the culprit. I ignored the transport port in envoy by setting

```auto
      annotations:
        "traffic.sidecar.istio.io/excludeInboundPorts": "9300"
        "traffic.sidecar.istio.io/excludeOutboundPorts": "9300"

```

Now the cluster connection is stable.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 14, 2020, 8:23am UTC](https://discuss.elastic.co/t/elasticsearch-on-kubernetes-losing-cluster-connection-every-hour/237086/3 "2020-07-14T08:23:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.