# Shard rebalancing is slow after network failure on any node

**URL:** https://discuss.elastic.co/t/shard-rebalancing-is-slow-after-network-failure-on-any-node/164716
**Category:** Elasticsearch
**Created:** [January 17, 2019, 11:49pm UTC](https://discuss.elastic.co/t/shard-rebalancing-is-slow-after-network-failure-on-any-node/164716 "2019-01-17T23:49:21Z")
**Posts on this page:** 1
**Showing post:** 6

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 19, 2019, 10:06am UTC](https://discuss.elastic.co/t/shard-rebalancing-is-slow-after-network-failure-on-any-node/164716/6 "2019-01-19T10:06:07Z")

</div>

> [@Ninad\_Pradhan](#):
>
> Its cat /proc/sys/net/ipv4/tcp\_retries2  
> 15  
> cat /proc/sys/net/ipv4/tcp\_syn\_retries  
> 6  
> yes transport.tcp.connect\_timeout is 30s

I think these settings are too high. In particular if `/proc/sys/net/ipv4/tcp_retries2` is 15 then it will take well over a minute to detect a dropped connection, during which time all sorts of other requests will be piling up in queues and generally causing trouble. If you reduce this setting to something more reasonable ([Red Hat say to reduce it to 3 in a HA situation](https://access.redhat.com/solutions/726753)) then the initial connection failure will be picked up much quicker.

That should be enough for cases where you disconnect a node that isn't the elected master. However if you disconnect the master node then a new master will be elected, and this initial election involves trying to reconnect to the disconnected node, which times out after `transport.tcp.connect_timeout`, and this happens twice, so with your settings that election takes at least another minute. I think that `30s` for the connect timeout is too long for many situations. It's certainly far too long for node-to-node connections, but unfortunately today Elasticsearch doesn't allow setting different timeouts for different kinds of connection so you can only change it for every outbound connection.

Could you reduce these settings to something more appropriate and re-run your experiment? If it is still taking longer than you expect then it would be useful if you could share the full logs from the master node for the duration of the outage so we can start to look at what else is taking so long.

> [@Ninad\_Pradhan](#):
>
> I see a message instantly suggesting the network failure was recognized in-fact.

The message you quote indicates that a single stats request timed out, but Elasticsearch cannot tell if this is because of a network issue or because the node was busy (e.g. doing GC) so it doesn't trigger any further actions. The most reliable way to get the cluster to react to a network partition is to drop a connection, and reducing `tcp_retries2` is a good way to do that.

---

_[View the full topic](https://discuss.elastic.co/t/shard-rebalancing-is-slow-after-network-failure-on-any-node/164716)._