# Nodes randomly disconnected from the ES cluster **URL:** https://discuss.elastic.co/t/nodes-randomly-disconnected-from-the-es-cluster/23028 **Category:** Elasticsearch **Created:** [April 1, 2015, 6:45am UTC](https://discuss.elastic.co/t/nodes-randomly-disconnected-from-the-es-cluster/23028 "2015-04-01T06:45:41Z") **Posts on this page:** 1 **Showing post:** 6
### Author: ![Tomer\_Levy](https://avatars.discourse-cdn.com/v4/letter/t/85e7bf/32.png) [@Tomer\_Levy](https://discuss.elastic.co/u/Tomer_Levy) #### Post date: [April 8, 2015, 9:33am UTC](https://discuss.elastic.co/t/nodes-randomly-disconnected-from-the-es-cluster/23028/6 "2015-04-08T09:33:35Z")
Link below seems like a good direction to solve the problem > **[Bug #1317811 “Dropped packets on EC2, “xen\_netfront: xennet: skb...” : Bugs :...](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811)** > > Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: > >     xen\_netfront: xennet: skb rides the rocket: 19 slots > > Each of these messages corresponds to a dropped TX packet, and eventually causes our... [image: photo] _Tomer Levy_ CEO, Co-Founder, [Logz.io](http://Logz.io) p:+972-544235023 | [e:tomer@logz.io](mailto:e:tomer@logz.io) | w:on.logz.io/1C2UlMi | a: +1-617-314-3318 [http://il.linkedin.com/pub/tomer-levy/1/950/360](http://il.linkedin.com/pub/tomer-levy/1/950/360) [http://twitter.com/tomerlevy](http://twitter.com/tomerlevy) Get a signature like this: [http://ws-stats.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS8/dXRtX3NvdXJjZT1leHRlbnNpb24mdXRtX21lZGl1bT1lbWFpbCZ1dG1fY2FtcGFpZ249cHJvbW9fNDUiLCAiZSI6ICJwcm9tb180NV9jbGljayJ9](http://ws-stats.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS8/dXRtX3NvdXJjZT1leHRlbnNpb24mdXRtX21lZGl1bT1lbWFpbCZ1dG1fY2FtcGFpZ249cHJvbW9fNDUiLCAiZSI6ICJwcm9tb180NV9jbGljayJ9) Click here! [http://ws-stats.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS8/dXRtX3NvdXJjZT1leHRlbnNpb24mdXRtX21lZGl1bT1lbWFpbCZ1dG1fY2FtcGFpZ249cHJvbW9fNDUiLCAiZSI6ICJwcm9tb180NV9jbGljayJ9](http://ws-stats.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS8/dXRtX3NvdXJjZT1leHRlbnNpb24mdXRtX21lZGl1bT1lbWFpbCZ1dG1fY2FtcGFpZ249cHJvbW9fNDUiLCAiZSI6ICJwcm9tb180NV9jbGljayJ9) On Tue, Apr 7, 2015 at 9:19 AM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > When you see this, can you check if \_cat/indices > ​,​ > \_cat/shards > ​and​ > \_cat/nodes > ​ return a request?​ > > On 7 April 2015 at 15:33, Tomer Levy [tomer.levy9@gmail.com](mailto:tomer.levy9@gmail.com) wrote: > > > We're experiencing a similar issue with one of our clusters on EC2 which > > was running 1.4.4 and it still happens after upgrade 1.5.0. We see "Master > > left" messages randomly happen and then reconnect after a couple of > > minutes. We have 4 data nodes and 3 master nodes (and a few client nodes). > > > > master left (reason = do not exists on master, act as master failure) > > > > Any thoughts? > > > > On Wednesday, April 1, 2015 at 10:13:35 AM UTC+3, Anil Karaka wrote: > > > > > I also use amazon, aws cloud plugin and discover my nodes based on the > > > security group.. > > > > > > should I instead change it to unicast discovery? > > > > > > On Wednesday, April 1, 2015 at 12:15:41 PM UTC+5:30, Anil Karaka wrote: > > > > > > > I greped for "removed" in master node and these are the logs that I see. > > > > > > > > [2015-04-01 05:32:55,813][INFO][cluster.service] > > > > [ESBigNode3] removed {[ES30GBNode2][Yf8ODQh0TE2\_ > > > > 0hQ35Y0M\_w][ip-153-31-43-55][inet[/153.31.73.55:9300]],}, reason: > > > > zen-disco-node\_failed([ES30GBNode2][Yf8ODQh0TE2\_ > > > > 0hQ35Y0M\_w][ip-153-31-73-55][inet[/153.31.73.55:9300]]), reason > > > > transport disconnected > > > > [2015-04-01 05:33:02,048][INFO][cluster.service] > > > > [ESBigNode3] removed {[ES30GBNode1][0CRaC261RXy8JfGc1XNLZA][ip- > > > > 153-31-76-111][inet[/153.31.76.111:9300]],}, reason: > > > > zen-disco-node\_failed([ES30GBNode1][0CRaC261RXy8JfGc1XNLZA][ip- > > > > 153-31-36-101][inet[/153.31.36.101:9300]]), reason transport > > > > disconnected > > > > [2015-04-01 05:33:09,702][INFO][cluster.service] > > > > [ESBigNode3] removed {[ESBigNode5][PaNaDPwfSM- > > > > jUpGa8HQJmQ][esnode5][inet[/153.31.70.128:9300]],}, reason: > > > > zen-disco-node\_failed([ESBigNode5][PaNaDPwfSM- > > > > jUpGa8HQJmQ][esnode5][inet[/153.31.70.128:9300]]), reason transport > > > > disconnected > > > > [2015-04-01 05:33:13,964][INFO][cluster.service] > > > > [ESBigNode3] removed {[ESBigNode1][ihJU17ToQVit9BxNzQjhnQ][ > > > > esnode1][inet[/153.31.75.190:9300]],}, reason: zen-disco-node\_failed([ > > > > ESBigNode1][ihJU17ToQVit9BxNzQjhnQ][esnode1][inet[/153.31.35.190:9300]]), > > > > reason transport disconnected > > > > > > > > And in the data node, this is how the node leaving the cluster looks > > > > like in its log files. > > > > > > > > [2015-01-22 20:49:56,860][WARN][discovery.ec2] > > > > [ESBigNode1] master left (reason = do not exists on master, act as master > > > > failure), current nodes: {[ESBigNode2][zVdCNza9Qk-v- > > > > Usu66jcvw][ip-153-31-73-29][inet[/153.31.73.29:9300]],[ESBigNode4][- > > > > 8pj8n2sS5GB4XTIE0zudQ][ip-153-31-74-230][inet[/153.31.74. > > > > 230:9300]],[ESBigNode1][nU6bkV-SSb6rvLHsth9AQg][ip- > > > > 153-31-75-190][inet[/153.31.75.190:9300]],} > > > > > > > > That is 4 nodes leaving the 7 node cluster at at time.. and the cluster > > > > is in red state for few minutes, not just yellow state.. > > > > Although 4 nodes leaving the cluster is rare.. Single nodes leave the > > > > cluster very often. > > > > > > > > As discussed in this thread, [https://groups.google](https://groups.google). > > > > com/forum/#!msg/elasticsearch/ixoAF9Yur0E/CgX4Hbk1ynYJ I will change > > > > the discovery.zen.ping.timeout to 10sec, what else can I do. > > > > > > > > there is an older thread from 2012 that also suggests to change OS > > > > settings that deal with ipv4 TCP keep alive settings.. Do I also have to > > > > change this setting? [Redirecting to Google Groups](https://groups.google.com/forum/#!msg/) > > > > elasticsearch/c9JmaiVfBb0/9XZM6ZJpoBwJ > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/1aed1011-c01b-4228-9e16-479330895c96%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1aed1011-c01b-4228-9e16-479330895c96%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/1aed1011-c01b-4228-9e16-479330895c96%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1aed1011-c01b-4228-9e16-479330895c96%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/E-aGhovVTPI/unsubscribe](https://groups.google.com/d/topic/elasticsearch/E-aGhovVTPI/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8ZXfsc8GU%3DogwTVPF-PkYdpq6kf6j%3DLTRQ5XuLQRWe%3Dg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8ZXfsc8GU%3DogwTVPF-PkYdpq6kf6j%3DLTRQ5XuLQRWe%3Dg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8ZXfsc8GU%3DogwTVPF-PkYdpq6kf6j%3DLTRQ5XuLQRWe%3Dg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8ZXfsc8GU%3DogwTVPF-PkYdpq6kf6j%3DLTRQ5XuLQRWe%3Dg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKf%2BfgcL5ZkreXRPArQ9fqNVTR6-bxp2FTPzsV61cdgziFg8jg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKf%2BfgcL5ZkreXRPArQ9fqNVTR6-bxp2FTPzsV61cdgziFg8jg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). --- _[View the full topic](https://discuss.elastic.co/t/nodes-randomly-disconnected-from-the-es-cluster/23028)._