# Very regular disconnect and recover - every 2 hours **URL:** \ **Category:** Elasticsearch\ **Created:** [March 30, 2015, 2:32pm UTC](https://discuss.elastic.co/t/very-regular-disconnect-and-recover-every-2-hours/22982 "2015-03-30T14:32:09Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![andrassy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrassy/32/32783_2.png) [@andrassy](https://discuss.elastic.co/u/andrassy)\ **Post date:** [March 30, 2015, 2:32pm UTC](https://discuss.elastic.co/t/very-regular-disconnect-and-recover-every-2-hours/22982/1 "2015-03-30T14:32:09Z")
Hi, I have two independent clusters running across more or less the same machines. They're split across a pretty high bandwidth and relatively low latency VPN link. One cluster is running v1.0.1 and seems to stay up all the time. The other cluster is currently running 1.4.4 (and was running 1.4.2 before that) and seems to disconnect like clockwork every two hours. The disconnect of the nodes on one side of the link is brief, they rejoin and the recovery proceeds as normal. Any ideas what might cause this? Could it be data related? The newer cluster has more indexes & shards than the old, but the co-ordinators (3 of / min master count 2) don't seem particularly stressed. Any thoughts on what, specifically to look for or whether any particular setting or code change might make the cluster more susceptible to disconnect when there's a minor / brief network connectivity blip? (and yes, I know multi-site isn't a recommended configuration - there are other challenges for us with the tribe node approach too, though ☹ ) Thanks in advance for any ideas or insight. N -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 31, 2015, 2:32am UTC](https://discuss.elastic.co/t/very-regular-disconnect-and-recover-every-2-hours/22982/2 "2015-03-31T02:32:47Z")
It's not the VPN reconnecting is it? On 31 March 2015 at 01:32, Neil Andrassy [neil.andrassy@thefilter.com](mailto:neil.andrassy@thefilter.com) wrote: > Hi, > > I have two independent clusters running across more or less the same > machines. They're split across a pretty high bandwidth and relatively low > latency VPN link. One cluster is running v1.0.1 and seems to stay up all > the time. The other cluster is currently running 1.4.4 (and was running > 1.4.2 before that) and seems to disconnect like clockwork every two hours. > The disconnect of the nodes on one side of the link is brief, they rejoin > and the recovery proceeds as normal. Any ideas what might cause this? Could > it be data related? The newer cluster has more indexes & shards than the > old, but the co-ordinators (3 of / min master count 2) don't seem > particularly stressed. Any thoughts on what, specifically to look for or > whether any particular setting or code change might make the cluster more > susceptible to disconnect when there's a minor / brief network connectivity > blip? > > (and yes, I know multi-site isn't a recommended configuration - there are > other challenges for us with the tribe node approach too, though ☹ ) > > Thanks in advance for any ideas or insight. > > N > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9R68ymc%2BgcX%2BBYopyz\_%2B%3D3MZ18fdBV-MRThHW\_BNA%2Bew%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9R68ymc%2BgcX%2BBYopyz_%2B%3D3MZ18fdBV-MRThHW_BNA%2Bew%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![andrassy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrassy/32/32783_2.png) [@andrassy](https://discuss.elastic.co/u/andrassy)\ **Post date:** [March 31, 2015, 5:57am UTC](https://discuss.elastic.co/t/very-regular-disconnect-and-recover-every-2-hours/22982/3 "2015-03-31T05:57:58Z")
It's probably something like that, but it only seems to be a problem with the more up to date version of ES. I'm keen to work out if there's a configuration option I can tweak in 1.4.4 to make ES more robust in this scenario or whether there's an issue around recovering dropped TCP connections between nodes in more recent versions. On Tuesday, 31 March 2015 03:33:18 UTC+1, Mark Walkom wrote: > It's not the VPN reconnecting is it? > > On 31 March 2015 at 01:32, Neil Andrassy \<[neil.a...@thefilter.com](mailto:neil.a...@thefilter.com) > \\> wrote: > > > Hi, > > > > I have two independent clusters running across more or less the same > > machines. They're split across a pretty high bandwidth and relatively low > > latency VPN link. One cluster is running v1.0.1 and seems to stay up all > > the time. The other cluster is currently running 1.4.4 (and was running > > 1.4.2 before that) and seems to disconnect like clockwork every two hours. > > The disconnect of the nodes on one side of the link is brief, they rejoin > > and the recovery proceeds as normal. Any ideas what might cause this? Could > > it be data related? The newer cluster has more indexes & shards than the > > old, but the co-ordinators (3 of / min master count 2) don't seem > > particularly stressed. Any thoughts on what, specifically to look for or > > whether any particular setting or code change might make the cluster more > > susceptible to disconnect when there's a minor / brief network connectivity > > blip? > > > > (and yes, I know multi-site isn't a recommended configuration - there are > > other challenges for us with the tribe node approach too, though ☹ ) > > > > Thanks in advance for any ideas or insight. > > > > N > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/793f13a8-9ca8-4d86-b194-47b4e9cd5125%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/793f13a8-9ca8-4d86-b194-47b4e9cd5125%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 31, 2015, 6:32am UTC](https://discuss.elastic.co/t/very-regular-disconnect-and-recover-every-2-hours/22982/4 "2015-03-31T06:32:06Z")
You can try winding out the timeouts, see > **[Node discovery | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-hosts-providers.html)** On 31 March 2015 at 16:57, Neil Andrassy [neil.andrassy@thefilter.com](mailto:neil.andrassy@thefilter.com) wrote: > It's probably something like that, but it only seems to be a problem with > the more up to date version of ES. I'm keen to work out if there's a > configuration option I can tweak in 1.4.4 to make ES more robust in this > scenario or whether there's an issue around recovering dropped TCP > connections between nodes in more recent versions. > > On Tuesday, 31 March 2015 03:33:18 UTC+1, Mark Walkom wrote: > > > It's not the VPN reconnecting is it? > > > > On 31 March 2015 at 01:32, Neil Andrassy [neil.a...@thefilter.com](mailto:neil.a...@thefilter.com) wrote: > > > > > Hi, > > > > > > I have two independent clusters running across more or less the same > > > machines. They're split across a pretty high bandwidth and relatively low > > > latency VPN link. One cluster is running v1.0.1 and seems to stay up all > > > the time. The other cluster is currently running 1.4.4 (and was running > > > 1.4.2 before that) and seems to disconnect like clockwork every two hours. > > > The disconnect of the nodes on one side of the link is brief, they rejoin > > > and the recovery proceeds as normal. Any ideas what might cause this? Could > > > it be data related? The newer cluster has more indexes & shards than the > > > old, but the co-ordinators (3 of / min master count 2) don't seem > > > particularly stressed. Any thoughts on what, specifically to look for or > > > whether any particular setting or code change might make the cluster more > > > susceptible to disconnect when there's a minor / brief network connectivity > > > blip? > > > > > > (and yes, I know multi-site isn't a recommended configuration - there > > > are other challenges for us with the tribe node approach too, though ☹ ) > > > > > > Thanks in advance for any ideas or insight. > > > > > > N > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/793f13a8-9ca8-4d86-b194-47b4e9cd5125%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/793f13a8-9ca8-4d86-b194-47b4e9cd5125%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/793f13a8-9ca8-4d86-b194-47b4e9cd5125%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/793f13a8-9ca8-4d86-b194-47b4e9cd5125%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_fbZXbP\_iY4ZpJODXPmumh15CntRT6S4HaJdrvqv593A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_fbZXbP_iY4ZpJODXPmumh15CntRT6S4HaJdrvqv593A%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![andrassy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrassy/32/32783_2.png) [@andrassy](https://discuss.elastic.co/u/andrassy)\ **Post date:** [March 31, 2015, 8:23am UTC](https://discuss.elastic.co/t/very-regular-disconnect-and-recover-every-2-hours/22982/5 "2015-03-31T08:23:43Z")
Both clusters have the following settings so, if that's related, I think there must be another contributing factor... "discovery.zen.fd.ping\_interval" : "1s", "discovery.zen.fd.ping\_timeout" : "60s", "discovery.zen.fd.ping\_retries" : "3", On 31 March 2015 at 07:32, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > You can try winding out the timeouts, see > [Node discovery | Elasticsearch Guide [8.11] | Elastic](http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#fault-detection) > > On 31 March 2015 at 16:57, Neil Andrassy [neil.andrassy@thefilter.com](mailto:neil.andrassy@thefilter.com) > wrote: > > > It's probably something like that, but it only seems to be a problem with > > the more up to date version of ES. I'm keen to work out if there's a > > configuration option I can tweak in 1.4.4 to make ES more robust in this > > scenario or whether there's an issue around recovering dropped TCP > > connections between nodes in more recent versions. > > > > On Tuesday, 31 March 2015 03:33:18 UTC+1, Mark Walkom wrote: > > > > > It's not the VPN reconnecting is it? > > > > > > On 31 March 2015 at 01:32, Neil Andrassy [neil.a...@thefilter.com](mailto:neil.a...@thefilter.com) > > > wrote: > > > > > > > Hi, > > > > > > > > I have two independent clusters running across more or less the same > > > > machines. They're split across a pretty high bandwidth and relatively low > > > > latency VPN link. One cluster is running v1.0.1 and seems to stay up all > > > > the time. The other cluster is currently running 1.4.4 (and was running > > > > 1.4.2 before that) and seems to disconnect like clockwork every two hours. > > > > The disconnect of the nodes on one side of the link is brief, they rejoin > > > > and the recovery proceeds as normal. Any ideas what might cause this? Could > > > > it be data related? The newer cluster has more indexes & shards than the > > > > old, but the co-ordinators (3 of / min master count 2) don't seem > > > > particularly stressed. Any thoughts on what, specifically to look for or > > > > whether any particular setting or code change might make the cluster more > > > > susceptible to disconnect when there's a minor / brief network connectivity > > > > blip? > > > > > > > > (and yes, I know multi-site isn't a recommended configuration - there > > > > are other challenges for us with the tribe node approach too, though ☹ ) > > > > > > > > Thanks in advance for any ideas or insight. > > > > > > > > N > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/793f13a8-9ca8-4d86-b194-47b4e9cd5125%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/793f13a8-9ca8-4d86-b194-47b4e9cd5125%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/793f13a8-9ca8-4d86-b194-47b4e9cd5125%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/793f13a8-9ca8-4d86-b194-47b4e9cd5125%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/FLsYRpcADEk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/FLsYRpcADEk/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_fbZXbP\_iY4ZpJODXPmumh15CntRT6S4HaJdrvqv593A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_fbZXbP_iY4ZpJODXPmumh15CntRT6S4HaJdrvqv593A%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_fbZXbP\_iY4ZpJODXPmumh15CntRT6S4HaJdrvqv593A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_fbZXbP_iY4ZpJODXPmumh15CntRT6S4HaJdrvqv593A%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Neil Andrassy | CTO | The Filter phone | +44 (0)1225 588 004 skype | andrassynp -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CABpTWLMn8k\_hbuRkh3s-ZmWqZd9s1POHzMQEuFFxWVugS5%3Dxng%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CABpTWLMn8k_hbuRkh3s-ZmWqZd9s1POHzMQEuFFxWVugS5%3Dxng%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![andrassy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrassy/32/32783_2.png) [@andrassy](https://discuss.elastic.co/u/andrassy)\ **Post date:** [April 1, 2015, 11:06am UTC](https://discuss.elastic.co/t/very-regular-disconnect-and-recover-every-2-hours/22982/6 "2015-04-01T11:06:33Z")
Further to this, the cluster that's failing has far more shards than the one that stays up. We have a number of daily date stamped indexes, each retaining 90 days of data amounting to 7500+ shards. Looking through github, it looks like some underlying changes might have impacted the scalability of clusters with larger numbers of shards (_[STORE] Cache fileLength for fully written files #9683_, _ES 1.4.2 random node disconnect #9212_ and \*[STORE] Add simple cache for StoreStats #9709 \*seem related). Looks like there are some improvements in 1.5.0 (particulalry #9709). We're in the process of testing the upgrade (although there's a regression too that's holding us up slightly - exists filters don't work on indexes created prior to 1.3.0 _Mappings: Fix \_field\_names to be disabled on pre 1.3.0 indexes #10268_ - fixed in 1.5.1 when it arrives). We have also attempted to reduce the number of shards where possible as a short term workaround. We also found that CPU usage is VERY high on the nodes that stayed in cluster during recovery - we've dropped concurrent shard recoveries to 1 (from the default of 3) to ease this situation. We're also considering weekly or even monthly rolling indexes to reduce shard count but that's a bigger code change in our application. Watch this space... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/830a91b3-ba2e-40b2-a166-8e0e14a4f99d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/830a91b3-ba2e-40b2-a166-8e0e14a4f99d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![andrassy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrassy/32/32783_2.png) [@andrassy](https://discuss.elastic.co/u/andrassy)\ **Post date:** [April 10, 2015, 10:35am UTC](https://discuss.elastic.co/t/very-regular-disconnect-and-recover-every-2-hours/22982/7 "2015-04-10T10:35:54Z")
Okay, looks like we had a couple of issues going on here, both of which I think we have resolved: 1. ES 1.4.4 was hammering CPU on the retained nodes during recovery, made worse by the fact that we'd dropped multiple nodes rather than just one. Reducing concurrent recoveries helped somewhat, but upgrading ES to 1.5.0 has returned CPU usage during recovery to a normal level. 2. The regular drop seems to be related to our VPN killing inactive TCP connections (as suspected). It seems like termination of any one of the many inter-node TCP connections results in all the other open and active connections also being dropped (including ping and cluster state). Maybe this behaviour has changed since the 1.0.\* versions? Altering the OS TCP keep-alive time down from 2 hours to 5 minutes has resolved the issue and our cluster now remains up and stable. Be warned, though, the keep-alive settings are at the OS (network interface) level and, on Windows at least, required a full server reboot. Hope this is useful to others and maybe can even go back into the pot for 2.0 resiliency (if it hasn't already) - better handling of dropped/terminated but re-establishable connections? Thanks all, N -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5911731c-2cea-40d5-95fe-f8fa1bcd5b28%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5911731c-2cea-40d5-95fe-f8fa1bcd5b28%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 10, 2015, 1:38pm UTC](https://discuss.elastic.co/t/very-regular-disconnect-and-recover-every-2-hours/22982/8 "2015-04-10T13:38:37Z")
I doubt this can be a task for 2.0 resiliency when the effect can be traced to a W$ndows TCP/IP socket default behavior together with a possible aggressive VPN implementation. ES is already using TCP keep-alive by default, see > **[Networking | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#tcp-settings)** What may be worth an examination is why a connection stays open and does no longer send keep alive packets, even though ES default configuration is telling another story. Network status dumps of open TCP connections could be helpful. Jörg On Fri, Apr 10, 2015 at 12:35 PM, Neil Andrassy \<[neil.andrassy@thefilter.com](mailto:neil.andrassy@thefilter.com) > wrote: > Okay, looks like we had a couple of issues going on here, both of which I > think we have resolved: > > 1. ES 1.4.4 was hammering CPU on the retained nodes during recovery, made > worse by the fact that we'd dropped multiple nodes rather than just one. > Reducing concurrent recoveries helped somewhat, but upgrading ES to 1.5.0 > has returned CPU usage during recovery to a normal level. > > 2. The regular drop seems to be related to our VPN killing inactive TCP > connections (as suspected). It seems like termination of any one of the > many inter-node TCP connections results in all the other open and active > connections also being dropped (including ping and cluster state). Maybe > this behaviour has changed since the 1.0.\* versions? Altering the OS TCP > keep-alive time down from 2 hours to 5 minutes has resolved the issue and > our cluster now remains up and stable. Be warned, though, the keep-alive > settings are at the OS (network interface) level and, on Windows at least, > required a full server reboot. > > Hope this is useful to others and maybe can even go back into the pot for > 2.0 resiliency (if it hasn't already) - better handling of > dropped/terminated but re-establishable connections? > > Thanks all, > > N > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/5911731c-2cea-40d5-95fe-f8fa1bcd5b28%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5911731c-2cea-40d5-95fe-f8fa1bcd5b28%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/5911731c-2cea-40d5-95fe-f8fa1bcd5b28%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5911731c-2cea-40d5-95fe-f8fa1bcd5b28%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEj%3Di6rNF1kysaks5udsnxuBtzAf10Z5A3RUguOz\_DSgg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEj%3Di6rNF1kysaks5udsnxuBtzAf10Z5A3RUguOz_DSgg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:20am UTC](https://discuss.elastic.co/t/very-regular-disconnect-and-recover-every-2-hours/22982/9 "2017-07-06T00:20:23Z")