# ES 1.4.2 random node disconnect
**URL:** https://discuss.elastic.co/t/es-1-4-2-random-node-disconnect/21552
**Category:** Elasticsearch
**Created:** [January 9, 2015, 5:55am UTC](https://discuss.elastic.co/t/es-1-4-2-random-node-disconnect/21552 "2015-01-09T05:55:49Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Revan007](https://discuss.elastic.co/u/Revan007)
#### Post date: [January 9, 2015, 5:55am UTC](https://discuss.elastic.co/t/es-1-4-2-random-node-disconnect/21552/1 "2015-01-09T05:55:49Z")
Hey,
I am having trouble for some while. I am getting random node disconnects
and I cannot explain why.
There is no increase in traffic ( search or index ) when this is happening
, it feels so random to me .
I first thought it could be the aws cloud plugin so I removed it and used
unicast and pointed directly to my nodes IPs but that didn't seem to be the
problem .
I changed the type of instances, now m3.2xlarge, added more instances, made
so much modifications in ES yml config and still nothing .
Changed java oracle from 1.7 to 1.8 , changed CMS collector to G1GC and
still nothing .
I am out of ideas ... how can I get more info on what is going on ?
Here are the logs I can see from master node and the data node
[http://pastebin.com/GhKfRkaa](http://pastebin.com/GhKfRkaa)
Current config:
6 m3.x2large, 1 master, 5 data nodes.
414 indices, index/day
7372 shards. 9 shards, 1 replica per index
208 million documents, 430 GB
15 gb heap size allocated per node
ES 1.4.2
Current yml config here :
[http://pastebin.com/Nmdr7F6J](http://pastebin.com/Nmdr7F6J)
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [January 10, 2015, 12:49pm UTC](https://discuss.elastic.co/t/es-1-4-2-random-node-disconnect/21552/2 "2015-01-10T12:49:55Z")
If you see
cluster:monitor/nodes/stats[n]] request\_id [82300775] timed out after
[15000ms]
in the logs, you have a monitor tool running that can not complete requests
because it takes longer than 15 seconds to traverse all the data folders on
all the nodes.
There are a number of methods to reduce disk traversal time in the data
folders:
- switch off monitoring (not really helpful) or reduce monitor interval
(maybe helpful, maybe not)
- increase stats request timeout (if monitor tools allow this but this does
not solve the cause of the problem)
- monitor only an index subset of your cluster (monitor tools usually do
not have this option)
- reduce number of segments per node -\> either by optimizing indices or
adding nodes
- wait for a fix in a future ES release
Have you counted the total number of segments? If the number is high, did
you run \_optimize with max\_num\_segments on your indices to reduce the
number of segments?
Jörg
On Fri, Jan 9, 2015 at 6:55 AM, Revan007 [dragosr@pionix.ro](mailto:dragosr@pionix.ro) wrote:
> Hey,
>
> I am having trouble for some while. I am getting random node disconnects
> and I cannot explain why.
> There is no increase in traffic ( search or index ) when this is happening
> , it feels so random to me .
> I first thought it could be the aws cloud plugin so I removed it and used
> unicast and pointed directly to my nodes IPs but that didn't seem to be the
> problem .
> I changed the type of instances, now m3.2xlarge, added more instances,
> made so much modifications in ES yml config and still nothing .
> Changed java oracle from 1.7 to 1.8 , changed CMS collector to G1GC and
> still nothing .
>
> I am out of ideas ... how can I get more info on what is going on ?
>
> Here are the logs I can see from master node and the data node
> [http://pastebin.com/GhKfRkaa](http://pastebin.com/GhKfRkaa)
>
> Current config:
>
> 6 m3.x2large, 1 master, 5 data nodes.
> 414 indices, index/day
> 7372 shards. 9 shards, 1 replica per index
> 208 million documents, 430 GB
> 15 gb heap size allocated per node
> ES 1.4.2
>
> Current yml config here :
> [http://pastebin.com/Nmdr7F6J](http://pastebin.com/Nmdr7F6J)
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHv0wxNXq\_nJrj5ByxrpZmwbdiKmMUbu4YYfjuGM5XkAA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHv0wxNXq_nJrj5ByxrpZmwbdiKmMUbu4YYfjuGM5XkAA%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Revan007](https://discuss.elastic.co/u/Revan007)
#### Post date: [January 10, 2015, 12:56pm UTC](https://discuss.elastic.co/t/es-1-4-2-random-node-disconnect/21552/3 "2015-01-10T12:56:52Z")
Hey, thank you for answering, I am using Marvel latest version.
Here is more info about the problem :
>
>
> Hey,
>
> I am having trouble for some while. I am getting random node disconnects a…nd I cannot explain why.
> There is no increase in traffic ( search or index ) when this is happening , it feels so random to me .
> I first thought it could be the aws cloud plugin so I removed it and used unicast and pointed directly to my nodes IPs but that didn't seem to be the problem .
> I changed the type of instances, now m3.2xlarge, added more instances, made so much modifications in ES yml config and still nothing .
> Changed java oracle from 1.7 to 1.8 , changed CMS collector to G1GC and still nothing .
>
> I am out of ideas ... how can I get more info on what is going on ?
>
> Here are the logs I can see from master node and the data node
> http://pastebin.com/GhKfRkaa
On Saturday, January 10, 2015 at 2:50:02 PM UTC+2, Jörg Prante wrote:
> If you see
>
> cluster:monitor/nodes/stats[n]] request\_id [82300775] timed out after
> [15000ms]
>
> in the logs, you have a monitor tool running that can not complete
> requests because it takes longer than 15 seconds to traverse all the data
> folders on all the nodes.
>
> There are a number of methods to reduce disk traversal time in the data
> folders:
>
> - switch off monitoring (not really helpful) or reduce monitor interval
> (maybe helpful, maybe not)
>
> - increase stats request timeout (if monitor tools allow this but this
> does not solve the cause of the problem)
>
> - monitor only an index subset of your cluster (monitor tools usually do
> not have this option)
>
> - reduce number of segments per node -\> either by optimizing indices or
> adding nodes
>
> - wait for a fix in a future ES release
>
> Have you counted the total number of segments? If the number is high, did
> you run \_optimize with max\_num\_segments on your indices to reduce the
> number of segments?
>
> Jörg
>
> On Fri, Jan 9, 2015 at 6:55 AM, Revan007 \\>
> wrote:
>
> > Hey,
> >
> > I am having trouble for some while. I am getting random node disconnects
> > and I cannot explain why.
> > There is no increase in traffic ( search or index ) when this is
> > happening , it feels so random to me .
> > I first thought it could be the aws cloud plugin so I removed it and used
> > unicast and pointed directly to my nodes IPs but that didn't seem to be the
> > problem .
> > I changed the type of instances, now m3.2xlarge, added more instances,
> > made so much modifications in ES yml config and still nothing .
> > Changed java oracle from 1.7 to 1.8 , changed CMS collector to G1GC and
> > still nothing .
> >
> > I am out of ideas ... how can I get more info on what is going on ?
> >
> > Here are the logs I can see from master node and the data node
> > [http://pastebin.com/GhKfRkaa](http://pastebin.com/GhKfRkaa)
> >
> > Current config:
> >
> > 6 m3.x2large, 1 master, 5 data nodes.
> > 414 indices, index/day
> > 7372 shards. 9 shards, 1 replica per index
> > 208 million documents, 430 GB
> > 15 gb heap size allocated per node
> > ES 1.4.2
> >
> > Current yml config here :
> > [http://pastebin.com/Nmdr7F6J](http://pastebin.com/Nmdr7F6J)
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fca12a91-07f6-4152-a4e4-97098e68fd0e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fca12a91-07f6-4152-a4e4-97098e68fd0e%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Revan007](https://discuss.elastic.co/u/Revan007)
#### Post date: [January 10, 2015, 1:18pm UTC](https://discuss.elastic.co/t/es-1-4-2-random-node-disconnect/21552/4 "2015-01-10T13:18:18Z")
The thing is I don't think is the monitor plugin. When this happens, my
node gets disconnected and the cluster goes into yellow state till it
recovers . I am using curator optimize , it is set to 2 segments for
indices older than 2 days .
On Saturday, January 10, 2015 at 2:56:52 PM UTC+2, Revan007 wrote:
> Hey, thank you for answering, I am using Marvel latest version.
>
> Here is more info about the problem :
>
> [ES 1.4.2 random node disconnect · Issue #9212 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/9212#issuecomment-69292232)
>
> On Saturday, January 10, 2015 at 2:50:02 PM UTC+2, Jörg Prante wrote:
>
> > If you see
> >
> > cluster:monitor/nodes/stats[n]] request\_id [82300775] timed out after
> > [15000ms]
> >
> > in the logs, you have a monitor tool running that can not complete
> > requests because it takes longer than 15 seconds to traverse all the data
> > folders on all the nodes.
> >
> > There are a number of methods to reduce disk traversal time in the data
> > folders:
> >
> > - switch off monitoring (not really helpful) or reduce monitor interval
> > (maybe helpful, maybe not)
> >
> > - increase stats request timeout (if monitor tools allow this but this
> > does not solve the cause of the problem)
> >
> > - monitor only an index subset of your cluster (monitor tools usually do
> > not have this option)
> >
> > - reduce number of segments per node -\> either by optimizing indices or
> > adding nodes
> >
> > - wait for a fix in a future ES release
> >
> > Have you counted the total number of segments? If the number is high, did
> > you run \_optimize with max\_num\_segments on your indices to reduce the
> > number of segments?
> >
> > Jörg
> >
> > On Fri, Jan 9, 2015 at 6:55 AM, Revan007 [dra...@pionix.ro](mailto:dra...@pionix.ro) wrote:
> >
> > > Hey,
> > >
> > > I am having trouble for some while. I am getting random node disconnects
> > > and I cannot explain why.
> > > There is no increase in traffic ( search or index ) when this is
> > > happening , it feels so random to me .
> > > I first thought it could be the aws cloud plugin so I removed it and
> > > used unicast and pointed directly to my nodes IPs but that didn't seem to
> > > be the problem .
> > > I changed the type of instances, now m3.2xlarge, added more instances,
> > > made so much modifications in ES yml config and still nothing .
> > > Changed java oracle from 1.7 to 1.8 , changed CMS collector to G1GC and
> > > still nothing .
> > >
> > > I am out of ideas ... how can I get more info on what is going on ?
> > >
> > > Here are the logs I can see from master node and the data node
> > > [http://pastebin.com/GhKfRkaa](http://pastebin.com/GhKfRkaa)
> > >
> > > Current config:
> > >
> > > 6 m3.x2large, 1 master, 5 data nodes.
> > > 414 indices, index/day
> > > 7372 shards. 9 shards, 1 replica per index
> > > 208 million documents, 430 GB
> > > 15 gb heap size allocated per node
> > > ES 1.4.2
> > >
> > > Current yml config here :
> > > [http://pastebin.com/Nmdr7F6J](http://pastebin.com/Nmdr7F6J)
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> > >
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4cc37e2e-4bbc-483d-bbbe-6cd0138d6689%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4cc37e2e-4bbc-483d-bbbe-6cd0138d6689%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:39am UTC](https://discuss.elastic.co/t/es-1-4-2-random-node-disconnect/21552/5 "2017-07-06T00:39:44Z")