# Cluster Becomes Unresponsive for 90 Sec After Data Node Leaves

**URL:** <https://discuss.elastic.co/t/cluster-becomes-unresponsive-for-90-sec-after-data-node-leaves/73788>\
**Category:** Elasticsearch\
**Created:** [February 3, 2017, 5:16am UTC](https://discuss.elastic.co/t/cluster-becomes-unresponsive-for-90-sec-after-data-node-leaves/73788 "2017-02-03T05:16:53Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Michael\_Sander](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_sander/32/838_2.png) [@Michael\_Sander](https://discuss.elastic.co/u/Michael_Sander)\
**Post date:** [February 3, 2017, 5:16am UTC](https://discuss.elastic.co/t/cluster-becomes-unresponsive-for-90-sec-after-data-node-leaves/73788/1 "2017-02-03T05:16:53Z")

</div>

Hi, I have a ES 5.2 cluster with 3 master nodes and a number of data nodes. I have Kibana/X-pack monitoring installed, but right now there are no indices or real data on this cluster (I'm just testing). I have been experimenting with adding and removing data nodes to see how the cluster reacts.

The problem is that often when I remove a data node, the entire cluster becomes unresponsive for about 90 seconds, I've seen it take a minute 40 seconds too. After that, things go back to normal. Is this normal behavior? If not, any insight on how this could occur (perhaps some timeout settings)? This is intermittent, sometimes it fixes itself immediately.

More details: I am running the ES instances within Docker instances, within a Kubernetes cluster. Each node only has a single Kubernetes Pod, and each Kubernetes Pod only has a single Docker container. This is all being run on Google Cloud platform. My `elasticsearch.yml` is below.

```auto
cluster:
  name: ${CLUSTER_NAME}
node:
  name: ${HOSTNAME}
  # Set to true/false depending on Dockerfile
  master: ${NODE_MASTER}
  data: ${NODE_DATA}
network.host:
  - _local_
  - _site_
path:
  data: /data/data
  logs: /data/log
http:
  enabled: ${HTTP_ENABLE}
  compression: true
  cors:
    enabled: false
cloud:
  kubernetes:
    service: elasticsearch-discovery
    namespace: es-da-cluster
discovery:
  type: kubernetes
  zen:
    minimum_master_nodes: ${NUMBER_MIN_MASTERS}
xpack:
  monitoring:
    enabled: true
  security:
    enabled: false
  graph:
    enabled: false
  watcher:
    enabled: false

```

My cluster logs when this occurs are [located on pastbin here](http://pastebin.com/raw/M65Da6Ar). I did a `tail -f` on these logs while it was occurring. While the cluster was unresponsive, there was no log output. The logs only got appended after the cluster came back to life roughly 90 seconds later.

And to further clarify, by "unresponsive", I mean making any call just hangs. Even if I have ssh-ed into a master node, just `curl localhost:9200` will hang for 90 seconds.

---

<div class="post-metadata">

**Author:** ![Michael\_Sander](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_sander/32/838_2.png) [@Michael\_Sander](https://discuss.elastic.co/u/Michael_Sander)\
**Post date:** [February 3, 2017, 7:30am UTC](https://discuss.elastic.co/t/cluster-becomes-unresponsive-for-90-sec-after-data-node-leaves/73788/2 "2017-02-03T07:30:03Z")

</div>

I figured it out myself. The Kubernetes discovery mechanism uses the zen unicast system under the hood. The [default fault detection requires 90 seconds](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#fault-detection) (30 second timeouts, 3 times). Just need to change the settings under `discovery.zen.fd`. Not sure I understand the defaults too well here.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 3, 2017, 7:30am UTC](https://discuss.elastic.co/t/cluster-becomes-unresponsive-for-90-sec-after-data-node-leaves/73788/3 "2017-03-03T07:30:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.