# Cluster recovery and reachability takes long time when master left

**URL:** https://discuss.elastic.co/t/cluster-recovery-and-reachability-takes-long-time-when-master-left/165747
**Category:** Elasticsearch
**Created:** [January 25, 2019, 10:24am UTC](https://discuss.elastic.co/t/cluster-recovery-and-reachability-takes-long-time-when-master-left/165747 "2019-01-25T10:24:07Z")
**Posts on this page:** 1
**Showing post:** 3

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 25, 2019, 2:34pm UTC](https://discuss.elastic.co/t/cluster-recovery-and-reachability-takes-long-time-when-master-left/165747/3 "2019-01-25T14:34:33Z")

</div>

This sounds similar to the discussion here:

> [@Shard rebalancing is slow after network failure on any node](https://discuss.elastic.co/t/shard-rebalancing-is-slow-after-network-failure-on-any-node/164716/6):
>
> I think these settings are too high. In particular if /proc/sys/net/ipv4/tcp\_retries2 is 15 then it will take well over a minute to detect a dropped connection, during which time all sorts of other requests will be piling up in queues and generally causing trouble. If you reduce this setting to something more reasonable ([Red Hat say to reduce it to 3 in a HA situation](https://access.redhat.com/solutions/726753)) then the initial connection failure will be picked up much quicker. That should be enough for cases where you disconnect a nod…

Reduce `net.ipv4.tcp_retries2` and the connection timeout and you should see improvements. Even 10-20 seconds sounds like a long time for the cluster to recover with those settings set appropriately, and I'd be interested to see logs from a recovery that did take that long.

Edit: it looks like you have reduced the ping timeout to `4s` in Elasticsearch, which will help it detect the connection drop a little quicker but can harm your cluster stability since it will remove nodes from the cluster if they pause for a few seconds of GC. It's much better to detect the connection drop with `net.ipv4.tcp_retries2` since this is independent of GC.

---

_[View the full topic](https://discuss.elastic.co/t/cluster-recovery-and-reachability-takes-long-time-when-master-left/165747)._