# Random data node disconnections on AWS
**URL:** https://discuss.elastic.co/t/random-data-node-disconnections-on-aws/75032
**Category:** Elasticsearch
**Created:** [February 14, 2017, 12:38pm UTC](https://discuss.elastic.co/t/random-data-node-disconnections-on-aws/75032 "2017-02-14T12:38:53Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@anishm](https://discuss.elastic.co/u/anishm)
#### Post date: [February 14, 2017, 12:38pm UTC](https://discuss.elastic.co/t/random-data-node-disconnections-on-aws/75032/1 "2017-02-14T12:38:53Z")
We are running a Elasticsearch 5.0.2 cluster of 10 m4.2xlarge data nodes with 5 m4.xlarge master nodes and currently hosting around 21TB of data. The regular ingest rate goes up to 4000/second and search rate around 300/second. The indices are multi tenant monthly with most queries going to the indices of current month. For discovery, we are using the discovery-ec2 plugin.
The cluster is stable for most amount of time, but we regularly see "master\_left" exceptions coming from random data nodes. Here is an example message
> [p-elasticsearch-data-10] master\_left [{p-elasticsearch-master-4}{3E2FXGGaQkCVva\_P\_y-KHw}{Xp9lS3j1TT-xKnGIiWQMzQ}{10.0.19.220}{10.0.19.220:9300}{aws\_availability\_zone=us-east-1d, aws\_availibility\_zone=us-east-1d}], reason [failed to ping, tried [3] times, each with maximum [30s] timeout]
Looking at some previous discussions in the forum I found that some changes had to be made to the sysctl variables. I made those changes:
> net.ipv4.tcp\_keepalive\_time: 600
> net.ipv4.tcp\_keepalive\_intvl: 60
> net.ipv4.tcp\_keepalive\_probes: 3
However, the situation is not resolving. This is resulting in a massive shard reallocation at random hours of day slowing our queries down. Are there any more configuration properties I am missing?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 14, 2017, 12:38pm UTC](https://discuss.elastic.co/t/random-data-node-disconnections-on-aws/75032/2 "2017-03-14T12:38:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.